Summary

OCI support would allow users to package and distribute terraform modules and providers as oci-archives that can be tagged and distributed on any oci-compliant container registry such as AWS ECR and GitHub packages.

Problem Statement

Tofu currently relies on a limited number of registry implementations to host and manage modules and providers. This creates challenges when it comes to implementing a strategy for module version management across large projects and organizations

Supporting the oci archive format offers the user the ability to package, tag and distribute their modules via a wider variety of registries. Tagging should provide a simple and effective way of versioning modules and managing them across organizations

User-facing description

Provider setup

terraform {
	required_providers {
		hashicups = {
			source = "oci://ghcr.io/myorg/hashicups"
			version = "3.1.0"
		}
	}
}

Module setup

module "s3_bucket" {
	source = "oci://ghcr.io/myorg/aws-s3-bucket"
	version = "1.0.0"
	bucket_name = "test_bucket"
}

New oci commands would be added to support the feature

package

cmd:
tofu oci package --tag <uri>/<name>:<semver> --module ./modules/org_s3_bucket
Description:
To package the module the user must provide a tag and path to the file or directory. module and provider flags are mutually exclusive and are used to identify the type of package that is to be created. module expects a directory path while providers expects a path to the provider binary.
Flags:

• --module: path to module directory
• --provider: path to provider binary
• --tag: uri for remote file repository

Pull

cmd:
tofu oci pull <tag>
Description:
Pull the image from a remote repository and and decompress the module into .terraform/modules manually

Login

cmd:
tofu oci login --username $USER --password $PASSWORD <uri>
Description:
Saves the credentials as base64 encoded string of $USER:$PASSWORD
Flags:

• --username: username for remote repository
• --password: password for remote repository
• --password-stdin: password value piped in from standard in

Push

cmd:
tofu oci push <tag>
Description:
Pushes the tagged package to the remote repository.

Technical Description

Packaging

In order to package our module we must compress the module directory into a tarball that will represent a layer or blob in our image. When compressing your module directory, the parent directory of the module should be included in the archive and its name should conform to the name set when tagging the package.

This is to ensure that when we decompress the blob to .terraform/modules/ it should create a referable path such as .terraform/modules/aws-s3-bucket in the example given above.

All blobs should use the sha256 hash as their filename and be located within /blobs/sha256/

each package should be structured as seen below.

blobs/sha256/
	- 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
	- 2fa21be67085dac1155fd09abfbcce31c9923ba3d1273c8046f9e50cfa7707b4 # This is your module 
	- e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f # This is your manifest
index.json

packages must contain:
- index at the root of the package named index.json
- At least one manifest defined within index.json and stored as a blob
- config stored as a blob
- tarball with the contents of given module stored as a blob

Index

Media-Type: application/vnd.oci.image.index.v1+json

The index file allows you to declare multiple manifests and you can use architecture to identify the correct manifest. This can be used to differentiate between provider versions depending on architecture Further Info can be found HERE

Example Index

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 7143,
      "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ],
}

Manifests

Media-Type: application/vnd.oci.image.manifest.v1+json

The manifest file is what's used to describe our image and is used to determine the location of blobs that the runtime must download. it should follow image-spec outlined HERE

Example Manifest

{
  "schemaVersion": "2",
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.tofu.module.v1.tar",
      "digest": "sha256:2fa21be67085dac1155fd09abfbcce31c9923ba3d1273c8046f9e50cfa7707b4",
      "size": 2130
    }
  ]
}

Media Types
application/vnd.oci.image.* is reserved for the spec and should not be extended. Instead we will define our own media types to describe modules and provider layers

• application/vnd.tofu.module.v1.tar
  • Primary module in package
• application/vnd.tofu.provider.v1.tar
  • layer containing provider
• application/vnd.tofu.config.v1+json
  • (Optional) describes metadata about the package

Config
Config file is a required part of the spec. In the example above we indicate that our blob is a file containing an empty JSON object {}. This could be utilized to facilitate more advanced features in future iterations

Registry Communication

All HTTP endpoints for registries can be found HERE and is essential reading if you are going to implement pull and push functionality

Authentication

Most registries use some form of basic auth for their authentication. when the user executes the login command. The login info should be encoded as a base64 string and stored as a token value within ~/.terraform.d/credentials.tfrc.json with a URI being used as the key. This token should then be used when making any of the repository operations.

Pulling

This functionality should be implemented as a new module getter. Call HEAD /v2/<name>/manifests/<tag> to check if the manifest for this tag exists. If so use a GET request to retrieve the file and iterate through the layers array to identify the digest of our module blob.

Using the digest we can call /v2/<name>/blobs/<digest> to download our compressed module. Its contents should be decompressed into ./.terraform/modules/<name>

Pushing

When pushing a new module we must always start by uploading all blob content. The monolithic approach to this is to iterate through each blob and POST the contents to POST /v2/<name>/blobs/uploads/?digest=<digest>. If successful it should return a 201 created status code

After uploading all the blobs, we must upload the manifest to PUT /v2/<name>/manifests/<ref> where ref represents the version tag we wish to assign.

Rationale and alternatives

There has been a number of alternative approaches suggested within #308 My aim with this feature is to maximize registry support with the least amount of divergence in tofu behaviour. I believe this gives us the best chance of success and the feature as a whole will drive tofu adoption.

Downsides

None identified.

Unresolved Questions

No response

Related Issues

• #308

Proof of Concept

TBC