Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for variables inside of required_providers blocks #1850

Open
dragonfleas opened this issue Jul 23, 2024 · 3 comments
Open

Allow for variables inside of required_providers blocks #1850

dragonfleas opened this issue Jul 23, 2024 · 3 comments
Labels
enhancement New feature or request pending-decision This issue has not been accepted for implementation nor rejected. It's still open to discussion.

Comments

@dragonfleas
Copy link

OpenTofu Version

Any version

Use Cases

The use cases for this are pretty broad but the primary focus is:

  • Ability to test modules easier by being able to pass in different required versions
  • Being able to easily test swap out providers if there is 3rd party provider competition in a certain space

Attempted Solutions

There has been a lot of discussion about this in Slack, there isn't really an alternative to this currently. If you want to do anything dynamically with a required_providers block, you have to do code-gen or HCL abstraction level golang libraries like hclwrite as described in this blog post.

So, the few alternatives that are...not great and a bit hacky are:

  • Do codegen with something like jq to generate providers in json
  • Use Terramate to create "stacks" that exist for various required provider configurations (The terramate team said terramate was not built for this use case explicitly)
  • Use hclwrite golang library as mentioned above

Proposal

This will allow you to do the following:

terraform {
    required_providers {
        aws = {
            source  = "hashicorp/aws"
            version = var.aws_version
        }
    }
}

References

@dragonfleas dragonfleas added enhancement New feature or request pending-decision This issue has not been accepted for implementation nor rejected. It's still open to discussion. labels Jul 23, 2024
@RLRabinowitz
Copy link
Contributor

Hello and thank you for this issue! I understand that this issue has already been discussed in Slack.
The core team will take this internally and discuss the issue, but this could take a little time.
Please bear with us while we get to it 🙏🏻

@apparentlymart
Copy link
Contributor

A somewhat-hairy issue to contend with for allowing dynamic substitution into source is that the source address of a provider persists from one round to the next in state snapshots, so someone using this would probably need to make sure that once they have used a particular source address to create some things they keep using exactly the same source address on all future runs of that state, to avoid creating a variation of the problem I described in #1896 (comment) where, as far as OpenTofu is concerned, the provider configuration that a resource instance is bound to has been removed while that resource instance still exists.


Allowing it for version seems less fraught though, since that's used by the provider installer (in tofu init) rather than by the runtime, and so has a similar set of constraints as for source and version in module blocks. The only special hazard I can think of there is that version in required_providers specifies one of potentially version constraints that are all considered together to select a single version that is suitable for all modules, which in particular means that:

  • If you use version = var.aws_version with var.aws_version set to an exact version constraint allowing only one version, you'll need to make sure that all of your modules all agree on setting version = var.aws_version, or at least make sure that all of your modules have inexact constraints describing version sets that all include the one specified in var.aws_version.
  • Each time you change var.aws_version you'll need to run tofu init -upgrade to tell OpenTofu it's allowed to select a new version, or else it'll complain that the version selection in the dependency lock file is not in the set of versions that all modules are compatible with.

I don't think either of these are insurmountable problems, but they do represent two additional hazards that have not been true for any of the locations where we've supported static evaluation so far, and I expect we'll need to think carefully about how OpenTofu could detect the above problems in a way that allows returning a relevant and actionable error message, rather than something highly confusing.

@dragonfleas
Copy link
Author

dragonfleas commented Oct 16, 2024

A somewhat-hairy issue to contend with for allowing dynamic substitution into source is that the source address of a provider persists from one round to the next in state snapshots, so someone using this would probably need to make sure that once they have used a particular source address to create some things they keep using exactly the same source address on all future runs of that state, to avoid creating a variation of the problem I described in #1896 (comment) where, as far as OpenTofu is concerned, the provider configuration that a resource instance is bound to has been removed while that resource instance still exists.

Allowing it for version seems less fraught though, since that's used by the provider installer (in tofu init) rather than by the runtime, and so has a similar set of constraints as for source and version in module blocks. The only special hazard I can think of there is that version in required_providers specifies one of potentially version constraints that are all considered together to select a single version that is suitable for all modules, which in particular means that:

* If you use `version = var.aws_version` with `var.aws_version` set to an exact version constraint allowing only one version, you'll need to make sure that _all_ of your modules all agree on setting `version = var.aws_version`, or at least make sure that all of your modules have inexact constraints describing version sets that all include the one specified in `var.aws_version`.

* Each time you change `var.aws_version` you'll need to run `tofu init -upgrade` to tell OpenTofu it's allowed to select a new version, or else it'll complain that the version selection in the dependency lock file is not in the set of versions that all modules are compatible with.

I don't think either of these are insurmountable problems, but they do represent two additional hazards that have not been true for any of the locations where we've supported static evaluation so far, and I expect we'll need to think carefully about how OpenTofu could detect the above problems in a way that allows returning a relevant and actionable error message, rather than something highly confusing.

Thanks for the insightful response here, I'd like to clarify from a Tofu DX perspective what kind of use case I'd see for this issue in particular.

In OpenTofu, there's an implied best practice for creating child modules, and that's if you are creating child modules (by this I am referring to modules that you intend to not be root modules, but library components that you intend to be consumed by others in root modules) it's implied in the documentation that your required_providers block should be using the minimum required version possible.

The problem with this approach is, even if you know the minimum feature set your module requires by the provider, there is no way to ascertain forward compatibility without being able to test new provider versions in an automated way.

It's very common that providers introduce breaking changes in minor versions, and strict semver adherence in the provider ecosystem is anemic to say the least. That being said, this would be a way for us to avoid having to create extremely verbose test scaffolding that instantiates child modules with a ton of different provider versions, and instead, have a single module where version is parameterized, and be able to create a CI jobs for each version of it's respective providers.

I think that aligns with the prospect of maybe an easier implementation of passing specifically a providers version as a variable, instead of the entire required_providers block, I'd even argue that might be a better abstraction since it prevents the edge case of removing a provider entirely inadvertently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request pending-decision This issue has not been accepted for implementation nor rejected. It's still open to discussion.
Projects
None yet
Development

No branches or pull requests

3 participants