Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to define the tunnel URL for Edge Agent deployments #6251

Closed
jamescarppe opened this issue Dec 10, 2021 · 19 comments
Closed
Labels
area/edge-agent kind/enhancement Applied to Feature Requests

Comments

@jamescarppe
Copy link
Member

jamescarppe commented Dec 10, 2021

Is your feature request related to a problem? Please describe.
When deploying an Edge Agent that is to communicate with a Portainer Server instance behind a reverse proxy (such as Traefik), the tunnel URL may need to be customized in order for the agent to correctly communicate with the server.

Consider the scenario where Portainer Server is running behind Traefik, with the following configuration (taken from our documentation):

    labels:
      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`portainer.yourdomain.com`)"
      - "traefik.http.routers.frontend.entrypoints=websecure"
      - "traefik.http.services.frontend.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend.service=frontend"
      - "traefik.http.routers.frontend.tls.certresolver=leresolver"

      # Edge
      - "traefik.http.routers.edge.rule=Host(`edge.yourdomain.com`)"
      - "traefik.http.routers.edge.entrypoints=websecure"
      - "traefik.http.services.edge.loadbalancer.server.port=8000"
      - "traefik.http.routers.edge.service=edge"
      - "traefik.http.routers.edge.tls.certresolver=leresolver"

In this instance, the Portainer Server is available at https://portainer.yourdomain.com whereas the Edge tunnel interface is at https://edge.yourdomain.com. At present, the tunnel URL is generated from the provided Portainer server URL, but without the protocol and with :8000 added to the end. This is provided to the Edge Agent via the base64 encoded join token. This with the above configuration fails to work, as port 8000 is not available via Traefik.

Currently, to make this work, a user would need to:

  1. Create their Edge Agent in Portainer Server using the Portainer URL but not deploy it
  2. Decode the base64 encoded join token (EDGE_KEY):
    https://portainer.yourdomain.com|portainer.yourdomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3
  3. Modify the second value to the correct URL:
    https://portainer.yourdomain.com|https://edge.yourdomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3
  4. Re-encode the modified token to URL-friendly base64
  5. Deploy the Edge Agent with the new join token in place of the one provided

Describe the solution you'd like
An additional (optional) field in the Edge Agent creation screen to provide an alternative tunnel URL for those that are required to do so with their configuration. When defined, this overrides the generated tunnel URL with the one provided.

Additional information
Thanks to portainer/portainer-compose#24 (comment) for helping to point me in the right direction for discovering this workaround.

@jamescarppe jamescarppe added kind/enhancement Applied to Feature Requests area/edge-agent labels Dec 10, 2021
@SvenDowideit
Copy link
Contributor

TBH, I'd like to separate the URL out from the connection key - because it can change - the original join token (containing the URL) should (imo) be ephemeral, and only relevant to bootstrapping the initial connection.

but you're right, we should managed to default that value all the time - but if there's a proxy setup, that default needs to be an editable setting. (I want to see it in the eng cfg screen for debug reasons, and to be able to edit it for IP change reasons)

@huib-portainer
Copy link
Contributor

Also note the duplicate: #4683

@pablosed
Copy link

Identical issue and also would like to see this implemented...

@yfhyou
Copy link

yfhyou commented Apr 19, 2022

Any tips for doing this in the meantime? Seems simple, but after decode, reencode I still get a lot of 2022/04/19 20:19:09 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] errors.

Currently, to make this work, a user would need to:

 Create their Edge Agent in Portainer Server using the Portainer URL but not deploy it
 Decode the base64 encoded join token:
 https://portainer.yourdomain.com|portainer.yourdomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3
 Modify the second value to the correct URL:
 https://portainer.yourdomain.com|https://edge.yourdomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3
 Re-encode the modified token to URL-friendly base64
 Deploy the Edge Agent with the new join token in place of the one provided

Nevermind. I forgot to allow the request past Authelia :) Working now!

@simonlock
Copy link

I too would like to see the explicit naming of the edge tunnel URL to be made possible; as I have also never got the edge agent to work using the method of base64 Decode/Encode when attempting to communicate with my
{{subdomain}}.{{domain}}.co.uk domain aka 4 level domain.

I've tried installing the edge agent on amd64, arm64, and arm32 architectures.

with url-safe encoding
2022/05/30 18:47:32 [INFO] [edge,registry] [message: Starting registry credential server]
2022/05/30 18:47:32 [INFO] [http] [server_addr: 172.17.0.2] [server_port: 9001] [secured: false] [api_version: 2.13.1] [message: Starting Agent API server]
2022/05/30 18:47:37 [ERROR] [edge] [message: an error occured during short poll] [error: short poll request failed]
2022/05/30 18:47:42 [ERROR] [edge] [message: an error occured during short poll] [error: short poll request failed]

without url-safe encoding
[ERROR] [main] [message: Unable to associate Edge key] [error: illegal base64 data at input byte 151

@huib-portainer
Copy link
Contributor

The edge key contains the following info encoded as base64:
portainer_instance_url|tunnel_server_addr|tunnel_server_fingerprint|endpoint_ID
e.g. https://portainer.yourdomain.co.uk|portainer.yourdomain.co.uk:8000|52:3f:78:a3:dd:67:4a:b3:9e:34:1a:87:6b:28:2b:dc|2
So I would expect that to work.

@simonlock
Copy link

I thought so too. However, swapping
https://portainer.yourdomain.co.uk|portainer.yourdomain.co.uk:8000|52:3f:78:a3:dd:67:4a:b3:9e:34:1a:87:6b:28:2b:dc|2
with
https://portainer.yourdomain.co.uk|https://edge.yourdomain.co.uk|52:3f:78:a3:dd:67:4a:b3:9e:34:1a:87:6b:28:2b:dc|2

I always get the behaviour that I explaned above. Could it be because I use a cloudflared tunnel (aka argo tunnel) with this domain? I wouldn't expect this to be the cause of the issue. But maybe a factor?

@pablosed
Copy link

I always get the behaviour that I explaned above. Could it be because I use a cloudflared tunnel (aka argo tunnel) with this domain? I wouldn't expect this to be the cause of the issue. But maybe a factor?

I have the exact same issue as you described, with the same setup ( portainer.example.com and edge.example.com) only difference is mine is behind a Traefik proxy (which makes zero difference to the other apps/services behind it). Same error message as well “short poll”.

I spent countless hours troubleshooting but could never get it working, same as you.

@simonlock
Copy link

Hi @dafinga, I am using a Traefik proxy too.

@pablosed
Copy link

Cool, @simonlock - well maybe that has something to do with it. Troubleshooting this further is above my pay grade. I have since torn down the entire setup regardless (although I would love a solution to this issue to surface as it would be the way I want to utilise the edge agent). Will continue to monitor this thread, and live in hope!

@AuthorityNull
Copy link

AuthorityNull commented Jul 15, 2022

Hope to see this addressed. Having to manually decode, edit and encode edge agent ID's for every new edge environment is incredibly inconvenient for an application that otherwise makes everything incredibly convenient.

This is also not documented very well and took me days of searching through threads and troubleshooting to figure out how to get edge agents behind a proxy to work.

@vflorian2000
Copy link

Why not just fix this for god's sake? Zero mention in the docs about having two different addresses in the key, no field to set edge address. If not for the solution suggested here on the issue post about decoding, correcting encoding, I wouldn't have found a solution. I've literally been looking for it for 2 and a half hours straight.

@AuthorityNull
Copy link

Why not just fix this for god's sake? Zero mention in the docs about having two different addresses in the key, no field to set edge address. If not for the solution suggested here on the issue post about decoding, correcting encoding, I wouldn't have found a solution. I've literally been looking for it for 2 and a half hours straight.

I feel you on this one. Took me and a colleague hours to figure out how to fix this.

@huib-portainer
Copy link
Contributor

You can give it a try by using the image portainerci/portainer-ee:pr1882.
Please let us know how that's working for you.
Note that this is a development build and should not be used in a production environment.

@octdanb
Copy link

octdanb commented Oct 9, 2022

@huib-portainer
I tried this on my dev box,
I have two rules with hosts, both with tls via letsencrypt
portainer.redacted.com -> portainer:9000
edge-portainer.redacted.com -> portainer:8000

Attempted to setup a new edge agent on a new ec2 instance, setting the new "Portainer tunnel server address" to https://edge-portainer.redacted.com

Used the new EDGE_ID, EDGE_KEY to configure the new edge agent.

it seems to try to connect over tcp, even if you specify a https address
What is the expected behavior here? Id like the option to do the tunnel through https

@huib-portainer
Copy link
Contributor

The edge connection will use TCP yes (by using the Chisel library).

@vflorian2000
Copy link

vflorian2000 commented Oct 9, 2022 via email

@acul009
Copy link

acul009 commented Feb 20, 2023

I'm kind of confused, I'd be happy if someone could help me out :)

The Issue is still open, but the Documentation found at https://docs.portainer.io/admin/settings/edge already shows and mentions "Portainer tunnel server address".

Is this feature available already?
Is this a CE or EE Feature?

If it's a EE feature, having the documentation mention that would be really nice.

The documentation for compose already has an issue using such a configuration.
portainer/portainer-compose#24

@jamescarppe
Copy link
Member Author

@acul009 Good catch, this is indeed a BE-only feature. I've updated the documentation to make this clearer.

In some cases we don't have automation attached to the closing of issues, and because we're human sometimes we miss closing them once a release goes out. In any case, I'll close this one now too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/edge-agent kind/enhancement Applied to Feature Requests
Projects
None yet
Development

No branches or pull requests

10 participants