1

My domain is:
sdxlive.com

I ran this command:
validating a SAN certificate using ansible acme_certificate module

It produced this output:
authorization:
challenges:
- error:
detail: '82.66.198.137: Fetching https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token: Error getting validation data'
status: 400
type: urn:ietf:params:acme:error:connection
status: invalid
token: token
type: http-01
url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/id/id2
validated: '2024-11-15T15:25:10Z'
validationRecord:
- addressUsed: 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
addressesResolved:
- 82.66.198.137
- 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
- 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
hostname: sdx1-kvm.sdxlive.com
port: '80'
url: http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
- addressUsed: 82.66.198.137
addressesResolved:
- 82.66.198.137
- 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
- 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
hostname: sdx1-kvm.sdxlive.com
port: '80'
url: http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
- addressUsed: 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
addressesResolved:
- 82.66.198.137
- 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
- 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
hostname: sdx1-kvm.sdxlive.com
port: '443'
url: https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
expires: '2024-11-22T15:24:56Z'
identifier:
type: dns
value: sdx1-kvm.sdxlive.com
status: invalid
uri: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id
identifier: dns:sdx1-kvm.sdxlive.com

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

From the official let's encrypt doc, "The HTTP-01 challenge can only be done on port 80".
However, when looking at the provided URL https://acme-v02.api.letsencrypt.org/acme/chall-v3/id/id2, I notice that Let's encrypt tries to contact the site over https:
"error": {
"type": "urn:ietf:params:acme:error:connection",
"detail": "82.66.198.137: Fetching https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token: Error getting validation data",
"status": 400
},

Of course, it fails because this server is not prepared to accept this request over https. However, it is fully functional over http and there is no redirection to https:
curl http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
... returns the expected file content

So, my conclusion is that the validation process has changed on your side to check over https first, which should be corrected. I do not see any other explanation.

2

Let's Encrypt issues roughly 5 million certs per day. If such a change / failure were to occur it would be a significant event :slight_smile: The LE internal alarms would be alerting staff I am sure.

The most likely explanation is that your system received the HTTP request but redirected it to HTTPS. LE will follow redirects and reports the last URL as the failing one. See: Challenge Types - Let's Encrypt

You have 3 IP addresses in your DNS. Your IPv4 address redirects HTTP requests to HTTPS. If there was a timeout reaching one of your IPv6 addresses LE will retry with IPv4. And, get redirected.

Are all of these IP able to reply properly to HTTP Challenge requests? LE prefers IPv6 so tries those first but it may use either one.

nslookup sdx1-kvm.sdxlive.com
Address: 82.66.198.137
Address: 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
Address: 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d

Example

curl -i4 http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/Test404
HTTP/1.1 301 Moved Permanently
Location: https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/Test404
4 Likes
3

Not at the time of the challenge.
Since my first post, I have removed the challenge URL setup and data, so now it is expected that the http request is redirected to https by design.
When the let's encrypt challenge nginx server setup was in place, I can assure you that there was no redirection to https.

TCP 80 port is open on all 3 IPv4 and IPv6 addresses.
I've just discovered that for some strange reason, one of the IPv6 addresses was on "dadfailed" state (2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d, not the other one), which is now corrected.

So, here's what must have happened:
LE tried 80 on 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d and failed due to an error on my side
LE did not try 80 on the 2 other IPs and went straight to https, which failed as expected.

LE should have tried 80 on the other 2 IPs before switching to 443.

You can check now the following:

nmap -p 80 -6 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
...
PORT   STATE SERVICE
80/tcp open  http

nmap -p 80 -6 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
nmap -p 80 82.66.198.137
4

I'm not sure the basis on which you think LE should have done this, but it doesn't. If you have multiple IPs in your DNS records, they all need to be able to respond appropriately to the challenge.

4 Likes
5

Yes, I see all 3 IP addresses now redirect HTTP Challenges to HTTPS. I don't know why you would want to do that rather than handling it right away but LE will follow the redirect.

No, LE will not arbitrarily try port 443 for an HTTP Challenge. That would be a violation of their requirements as a CA. It would only do that if redirected by your system.

And, it won't try each of your IP hoping to find one that works. It sends a single HTTP request to one of your IPv6 addresses. If it works it works. If not it fails. You can redirect that request as pointed out. And, LE will retry an IPv6 timeout with IPv4. But, only with the original HTTP request not after being redirected. (See: IPv6 Support - Let's Encrypt)

I don't think you appreciate how significant of a problem that would be if LE actually did that. It would be national news in our industry.

You are going to have to provide a better test case to reproduce that.

I have only seen you redirecting HTTP challenges to HTTPS. I think whatever glitch you saw was more likely due to the IPv6 "dadfail" you describe and LE doing its IPv4 retry.

Getting back to this point. How do you ensure LE will get a proper response regardless of which IPv6 address it tries?

3 Likes
6

So, I've just tried the challenge again, and the issue is still there despite:

The nginx setup is straightforward, only on 80 on purpose, with a few sanity checks:

server 
{
        listen 80;
        listen [::]:80;

        server_name sdx1-kvm.sdxlive.com;

	#########
	# Filters
	#########
        if ($request_method !~* ^(GET|HEAD|OPTIONS)$ )
        {
                return 444;
        }
        if ($request ~ \\x )
        {
                return 444;
        }

	##################################
        # Let's Encrypt Challenge Settings
	##################################
        # /.well-known/acme-challenge/*
	##################################
        location /
        {
                root <challenge_data_root_path>;
                try_files $uri =404;
        }

	##################
	# Logging Settings
	##################
	access_log      <log_path> main;
	error_log       <log_path> warn;
}

Another server runs on 443 and is unrelated to the LE challenge.

But, running the challenge keeps failing due to its access to https which is not defined for this purpose on this site:

    authorization:
      challenges:
      - error:
          detail: '82.66.198.137: Fetching https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token: Error getting validation data'
          status: 400
          type: urn:ietf:params:acme:error:connection
        status: invalid
        token: token
        type: http-01
        url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/id/id2
        validated: '2024-11-15T15:25:10Z'
        validationRecord:
        - addressUsed: 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
          addressesResolved:
          - 82.66.198.137
          - 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
          - 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
          hostname: sdx1-kvm.sdxlive.com
          port: '80'
          url: http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
        - addressUsed: 82.66.198.137
          addressesResolved:
          - 82.66.198.137
          - 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
          - 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
          hostname: sdx1-kvm.sdxlive.com
          port: '80'
          url: http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
        - addressUsed: 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
          addressesResolved:
          - 82.66.198.137
          - 2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d
          - 2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef
          hostname: sdx1-kvm.sdxlive.com
          port: '443'
          url: https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token
      expires: '2024-11-22T15:24:56Z'
      identifier:
        type: dns
        value: sdx1-kvm.sdxlive.com
      status: invalid
      uri: https://acme-v02.api.letsencrypt.org/acme/authz-v3/id
    identifier: dns:sdx1-kvm.sdxlive.com

This time, I have not removed the server setup on 80 so you can perform your own tests.
for instance, you might want to verify that http://sdx1-kvm.sdxlive.com/test.txt is accessible with no redirection to https.

7

I'm currently seeing some tests being made successfully by some readers, except for the ones trying inside /.well-known folder where the test.txt is not located.

8

I don't know what you mean. Can you explain?

Also, what is the actual URL of this:

2 Likes
9

I think he saw me doing a little poking around. I didn't see a redirection, and tested if there was a difference between the root text.txt and under the .well-known folder. I didn't see any redirections, but didn't test much. Didn't test different user agents either, but don't have time to dig in right now.

2 Likes
10

It's simple: some people reading this thread are trying successfully the URL http://sdx1-kvm.sdxlive.com/test.txt, as I see all the accesses in nginx logs, meaning they can access this resource without any redirection or other error.

The real URL has been obfuscated on purpose because it contains the real token value. I don't think you need it. The whole purpose of this post was to show that there was no redirection on my side despite numerous LE claims.

AFAIK, LE goes directly to 443 and does not even try 80, otherwise I would have seen these 80 successful or failed accesses in the logs.

11

You're not the only one. Some people try to access the token.

12

Yes, I was also one of them. I mean, you did ask us to look at it :slight_smile:

I disagree. It looks like some of the queries are being done by the ACME Client and not Let's Encrypt itself. It looks like the first was to an IPv6 address on port 80 followed by a port 80 to an IPv4 followed by an IPv6. We normally don't see that kind of series in the LE challenge error.

I wanted to see the actual LE challenge response.

2 Likes
13

I don't see any need for that.

1 Like
14

Here it is:

{
  "identifier": {
    "type": "dns",
    "value": "sdx1-kvm.sdxlive.com"
  },
  "status": "invalid",
  "expires": "2024-11-22T15:24:56Z",
  "challenges": [
    {
      "type": "http-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/id/id2",
      "status": "invalid",
      "validated": "2024-11-15T15:25:10Z",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "82.66.198.137: Fetching https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token: Error getting validation data",
        "status": 400
      },
      "token": "token",
      "validationRecord": [
        {
          "url": "http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token",
          "hostname": "sdx1-kvm.sdxlive.com",
          "port": "80",
          "addressesResolved": [
            "82.66.198.137",
            "2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d",
            "2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef"
          ],
          "addressUsed": "2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d"
        },
        {
          "url": "http://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token",
          "hostname": "sdx1-kvm.sdxlive.com",
          "port": "80",
          "addressesResolved": [
            "82.66.198.137",
            "2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d",
            "2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef"
          ],
          "addressUsed": "82.66.198.137"
        },
        {
          "url": "https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token",
          "hostname": "sdx1-kvm.sdxlive.com",
          "port": "443",
          "addressesResolved": [
            "82.66.198.137",
            "2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d",
            "2a01:e0a:81c:99c0:ab21:dfd4:b245:7ef"
          ],
          "addressUsed": "2a01:e0a:81c:99c0:558c:7a:5b5c:7b5d"
        }
      ]
    }
  ]
}

If the validationRecord actually reflects what has really been done on LE side and not just its intentions, it does show 2 accesses to 80 (1 in IPv6 and 1 in IPv4), but there are no record of those in my logs.

Could you give me the IPv6/IPv4 addresses of the server(s) validating the challenge?

15

Who knows what some people might do with it, otherwise why would some try to access its contents?

16

I have just seen a "successful" GET on 80 from an LE server, but:

  • with the wrong token
  • 8 mns after my last challenge run
"GET /.well-known/acme-challenge/wrong_token HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
17

No, I wanted to see the actual challenge URL. It has the error details as reported by Let's Encrypt. The messages shown by you look like they came from your ACME Client and not LE itself. Many ACME Clients do a "pre-check" so I'm trying to sort out exactly who sees what and when.

No, I can't. I am a volunteer. But, also Let's Encrypt checks from 5 different global locations in server farms where the IP addresses regularly change.

You should be able to see the requester IP in your log for any one specific request.

That doesn't sound like an actual LE request. What was the requester IP address? For one, LE wouldn't wait up to 8mins to try a request.

It also wasn't "successful" in that it got a 404 (Not Found).

Hiding info is just making this very difficult to debug. We are not asking for anything that needs to be secret.

3 Likes
18

You got it. I showed you exactly what the ``https://acme-v02.api.letsencrypt.org/acme/chall-v3/id/id2``` returned (minus the token value), believe it or not.

2600:3000:2710:300::87:

  • NetName: VIAWEST-IPV6-BLK-1
  • Organization: Flexential Colorado Corp. (VINS)

Hence the quotes, but the token looked like something real. I thought you guys were doing some tests. What I meant by "successful" was that at least, what appeared to be an LE server was able to contact an 80 port on my server.

19

I have just noticed another strange detail: the field challenges[0].error.detail:

"detail": "82.66.198.137: Fetching https://sdx1-kvm.sdxlive.com/.well-known/acme-challenge/token: Error getting validation data",

It shows that LE tried to access the port 443 at the IPv4 address, although the validationRecord shows that the 443 port has been or was supposed to be tried only on the IPv6 address.

It does not add up.

21

There is a known bug that shows the "wrong" IP with a mix of:
IPv6 and IPv4
IPv6 timeout failures
IPv4 redirects

Your latest series shows this sequence as did your original. I explained this timeout / retry earlier.

I don't have time to study this and/or reproduce it. Are you sure your ansible script isn't modifying the nginx config for these challenges?

Because the failing sequence looks just like earlier even though HTTP requests right now do not redirect.

Look carefully at the "Validation Record". This is the series

#1 See URL for HTTP(port 80) and addressUsed (IPv6)
#2 See URL for HTTP(port 80) and addressUsed (IPv4) (because #1 timed out)
#3 See URL for HTTPS(port 443) and addressUsed (IPv6) (because #2 redirected)

LE does not do IPv4 fallback after being redirected to HTTPS so challenge fails. See: IPv6 Support - Let's Encrypt

4 Likes