Hey @jjshanks thank you for this RFC. There are some concerns around the ability to hold a lock with OpenTofu exiting on backends, such as Consul as documented here. Could you expand your RFC and address how this would work with regards to specific backends?

Thanks for the link @janosdebugs. To summarize the other discussions the potential options are

1. Implement for all backends, this would require some backends (local, consul, etc.) to continue to run and enforce the lock.
   • For cases where the process needs to keep running could add a flag to make it explicit
2. Implement for some backends, this would require outputting some kind of error when unsupported backends are invoked

I don't have any experience using backends that aren't s3/dynamo so I'm not sure I'm the best person to say how other backends should behave. But my gut reaction would be to give an error for anything that needs to keep running for the lock, because users of those backends already don't use force-unlock.

I'll update the RFC to reflect the above for now until there is more feedback.

The current API for locking is the Lock method of statemgr.Locker.

Right now that doesn't have any way to distinguish between a lock that can safely be dropped automatically when the process exits vs. a lock that must survive beyond the end of the current process (or return an error if that's not possible).

It seems like the smallest change to the existing API would be to extend statemgr.LockInfo with a new field that serves as a hint to the implementation that it should prefer to fail if it can't promise to preserve the lock beyond the life of the process. The local and consul backends could then check for that hint and return an error if it's set, unless we can find some reasonable way for those backends to create a more persistent lock in future.

It's a awkward that the default behavior for an unmodified backend would be to disregard the new hint field and acquire a useless lock that will immediately be dropped, but thankfully since we have all of the backends in-tree right now we can make sure they all get updated to support the new hint field (if needed) as part of the same PR.

This is perhaps an example of something that would be a little harder if we waited until after #382, because we'd no longer be able to update all existing backends to support this new capability. (Though I think we could still do it, just that we'd need to take a different API design strategy such as having force-lock call an entirely new method that existing backends don't support at all, and then the default would be for the new command to not be supported until all backends have been updated, so there would likely always be backend implementations out there that could support force-lock but nonetheless haven't been updated to support it yet.)

Cross-posting to share the knowledge once force-unlock is implemented:)

If it's of any use to folks stumbling across this, here's a bash/zsh compatible interactive script to more easily unlock state. Could be updated for running in CI/CD in an automated fashion with some pre-validation - YMMY and use with due caution.

tf_unlock() {
    local LOCK_ID
    local ERROR_OUTPUT
    local LOCK_DETAILS

    # Capture both stdout and stderr
    ERROR_OUTPUT=$(terraform plan -json 2>&1)

    # Check if there's a lock error
    if [[ $ERROR_OUTPUT == *"Error acquiring the state lock"* ]]; then
        LOCK_DETAILS=$(echo "$ERROR_OUTPUT" | jq -r '.diagnostic.detail // empty' 2>/dev/null)
        if [[ -z "$LOCK_DETAILS" ]]; then
            LOCK_DETAILS=$(echo "$ERROR_OUTPUT" | grep -A10 "Lock Info:")
        fi
        LOCK_ID=$(echo "$LOCK_DETAILS" | awk '/ID:/ {print $2; exit}')

        echo "State is locked. Lock details:"
        echo "$LOCK_DETAILS"
        echo

        echo -n "Do you want to unlock this state? Type 'yes' to confirm: "
        read response
        if [[ "$response" == "yes" ]]; then
            echo "Attempting to unlock..."
            if terraform force-unlock --force "${LOCK_ID}"; then
                echo "Terraform state has been successfully unlocked!"
            else
                echo "Failed to unlock the state. Please check the error message above." >&2
                return 1
            fi
        else
            echo "Unlock cancelled."
            return 0
        fi
    elif [[ $ERROR_OUTPUT == *"Error:"* ]]; then
        echo "Error occurred while checking Terraform state:" >&2
        echo "$ERROR_OUTPUT" >&2
        return 1
    else
        echo "State is not locked. No action needed."
    fi
}

Originally posted by @darpham in hashicorp/terraform#30277 (comment)

@klongmitre tagging you here, this is the issue we've discussed at KubeCon recently.

If you are interested in this feature, please make sure to upvote the original issue, as we rely on the upvotes to determine community demand