Cloud unavailable: Make sure ssh key and username are valid #954

kkhamruddin-pk · 2019-05-21T08:14:22Z

Hi , I have installed mist-ce stable version (v4.0.3).

CONTAINER ID IMAGE dd1b0e526b59 mistce/nginx:v4-0-3 7d5d1c45b4c4 mistce/api:v4-0-3 6dd1586f6dce mistce/api:v4-0-3 5bbe4f886abd mistce/api:v4-0-3 c32c5a335a94 mistce/api:v4-0-3 b5daf14862d1 mistce/api:v4-0-3 d98e7b03302a mistce/api:v4-0-3 3b853be49828 mistce/api:v4-0-3 8aec2244da4b mistce/api:v4-0-3 afeef9376473 mistce/api:v4-0-3 ad935ab28015 mistce/logstash:v4-0-3 b1a5b8dfdc08 kibana:5.6.10 ebf6726a6f74 mistce/gocky:v4-0-3 e98fe60164d1 mistce/ui:v4-0-3 6e23a738f1f6 mist/mailmock b0a51d65f5a6 mistce/landing:v4-0-3 5868e3f44436 mist/swagger-ui 73ab7fa7a54d traefik:v1.5 c98633ccfa32 mongo:3.2 2858c4354836 mist/docker-socat c7d9dcca02c0 memcached dbf7cece144a elasticsearch:5.6.10 2728052bb940 rabbitmq:3.6.6-management 4335af4104d7 influxdb COMMAND CREATED STATUS PORTS NAMES
"nginx -g 'daemon of…" About an hour ago Up 16 minutes 0.0.0.0:80->80/tcp tmp_nginx_1
"/mist.api/bin/docke…" About an hour ago Up 15 minutes tmp_hubshell_1
"/mist.api/bin/docke…" About an hour ago Up 16 minutes tmp_scheduler_1
"/mist.api/bin/docke…" About an hour ago Up 15 minutes 80/tcp tmp_api_1
"/mist.api/bin/docke…" About an hour ago Up 16 minutes tmp_cilia_1
"/mist.api/bin/docke…" About an hour ago Up 16 minutes 8081/tcp tmp_sockjs_1
"/mist.api/bin/docke…" About an hour ago Up 16 minutes tmp_poller_1
"/mist.api/bin/docke…" About an hour ago Up 15 minutes tmp_celery-prefork_1
"/mist.api/bin/docke…" About an hour ago Up 16 minutes tmp_beat_1
"/mist.api/bin/docke…" About an hour ago Up 15 minutes tmp_celery-gevent_1
"/docker-entrypoint.…" About an hour ago Up 16 minutes tmp_logstash_1
"/docker-entrypoint.…" About an hour ago Up 16 minutes 5601/tcp tmp_kibana_1
"/go/src/github.com/…" About an hour ago Up 16 minutes 0.0.0.0:9096-9097->9096-9097/tcp tmp_gocky_1
"sh /entry.sh" About an hour ago Up 16 minutes 80/tcp tmp_ui_1
"twistd -n localmail…" About an hour ago Up 16 minutes 8025/tcp tmp_mailmock_1
"sh /entry.sh" About an hour ago Up 15 minutes 80/tcp tmp_landing_1
"sh /usr/share/nginx…" About an hour ago Up 16 minutes 8080/tcp tmp_swagger_1
"/traefik --accesslo…" About an hour ago Up 16 minutes 0.0.0.0:8040->80/tcp, 0.0.0.0:8041->8080/tcp tmp_traefik_1
"docker-entrypoint.s…" About an hour ago Up 16 minutes 27017/tcp tmp_mongodb_1
"socat TCP4-LISTEN:2…" About an hour ago Up 16 minutes 2375/tcp tmp_socat_1
"docker-entrypoint.s…" About an hour ago Up 16 minutes 11211/tcp tmp_memcached_1
"/docker-entrypoint.…" About an hour ago Up 15 minutes 9200/tcp, 9300/tcp tmp_elasticsearch_1
"docker-entrypoint.s…" About an hour ago Up 16 minutes 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp tmp_rabbitmq_1
"/entrypoint.sh infl…" About an hour ago Up 16 minutes 0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp tmp_influxdb_1

I am able to access the web interface, but not able to add the already installed KVM machine that runs ubuntu o.s.

I have added the root user ssh private key to the keys section. But when I tried to ADD a cloud am getting the below error on the webpage.

"Cloud unavailable: Make sure ssh key and username are valid"

Few additonal details:
uid=0(root) gid=0(root) groups=0(root),116(libvirtd)

From the api docker container:
/mist.api/bin # netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:41412 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1/uwsgi

From auth.log:
root@hostname:/tmp$ tail -f /var/log/auth.log
May 21 01:12:02 hil1c01 sshd[153568]: Did not receive identification string from 172.20.0.23
May 21 01:12:03 hil1c01 sshd[153571]: Connection closed by 172.20.0.23 port 47866 [preauth]

From docker inspect network:
[
{
"Name": "tmp_default",
"Id": "22eeb8ce4a987029dc49d70cd3546878af3744f091aefd172a400554f02be137",
"Created": "2019-05-20T23:51:00.870736946-07:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"2728052bb9400f77475d6327e6ff664f480567c6d1b21d81d00392c557e4a8cc": {
"Name": "tmp_rabbitmq_1",
"EndpointID": "e2cfb8efd306fe0ecac43ca07e408c2a15f93baee1be51e253ebdad29bb6b5f6",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
},

Can someone please help?

d-mo · 2019-05-21T17:47:35Z

Hi @kkhamruddin-pk,
unfortunately I've never witnessed such a situation myself. Can you please confirm that you can connect with ssh to the target host from within the api container? e.g.

docker-compose exec api sh
ssh -vvv -i id_rsa root@host

Replace "host" with the target hostname and "id_rsa" with the path of your private SSH key within the api container. You'll have to manually save it there first. The -vvv arg will cause ssh to output lots of logs that could be useful if it's not possible to connect using the target key.

Another thing to try is v4.1.0 which was just released. There have been some changes regarding SSH keys, however I don't really know if it will address your problem.

kkhamruddin-pk · 2019-05-22T04:30:09Z

@d-mo : Thanks for the quick response.

I copied the baremetal KVM machines root user id_rsa key to api container and ran the suggested command.

ssh -vvv -i id_rsa root@host

Am able to ssh to the baremetal kvm machine from api container. but noticed that it is still asking for password though I passed id_rsa private key ( is this expected?)and the error still persists.

Also, I have tried installing the v4.1.0 but no luck.

kkhamruddin-pk · 2019-05-29T00:00:44Z

Any update on this?

d-mo · 2019-05-29T23:57:59Z

Hi @kkhamruddin-pk,
unfortunately I haven't been able to replicate this issue. I was able to successfully add KVM hosts using my ssh keys in every one of my tests. If you can grant me temporary access to your Mist installation I think I'll be able to figure it out. If you'd like that please contact me at dimo (at) mist.io.

geraldpetz · 2019-07-22T12:11:04Z

@kkhamruddin-pk I have the exact same issue - could you solve the problem?

kkhamruddin-pk · 2019-07-23T08:45:44Z

@geraldpetz : I didn't find the solution for this after a lot of troubleshooting.

I just stopped using this tool.

d-mo · 2019-07-23T11:43:43Z

@geraldpetz @kkhamruddin-pk We're very interested in addressing this issue but we haven't been able to replicate it so far. If it's possible to provide me with temporary access to an environment where this can be replicated, then I'm quite confident that I can come up with a solution.

fruchtkeks · 2019-08-22T06:33:47Z

Hi @d-mo,
I'm running into the same issue. The connection from within the API container to my KVM host is working (without a fallback to password authentication). Unfortunately, I can't provide you with access to my system but would like to ask if you can suggest any steps for debugging the connection process myself?

d-mo · 2019-08-22T13:49:08Z

@fruchtkeks I don't understand how it's possible to be able to login using ssh as root with a passwordless key from the API container to the target machine, but at the same time not being able to add the host through the UI. If you could give me access I would add an import ipdb;ipdb.set_trace() statement to the respective API code and I would trace it line by line to figure out where exactly it fails. Unfortunately, I don't have access to such an environment and I haven't been able to replicate this issue anywhere else. If you can set up another environment where you can replicate the issue (e.g. on a public cloud VM) and give me temporary access to that then I would be happy to take a look.

A simpler thing you can try is to add the same host as an "Other server" cloud (instead of a KVM one) using the same SSH creds. If that also fails then that would indicate an SSH authentication issue. If it succeeds then the issue must be related to your KVM installation (e.g. inaccessible KVM API or wrong permissions).

fruchtkeks · 2019-08-22T15:37:45Z

Adding the KVM host as "Other Server" does work.
The API container throws the following error when trying to add the server as KVM host:

Traceback (most recent call last):
File "/mist.api/libcloud/libcloud/compute/drivers/libvirt_driver.py", line 158, in init
self.connection = libvirt.open(uri)
File "/usr/local/lib/python3.7/site-packages/libvirt.py", line 277, in open
if ret is None:raise libvirtError('virConnectOpen() failed')
libvirt.libvirtError: Cannot recv data: Load key "/tmp/tmpiqciln_8": invalid format

When opening the temporary key, it seems that the end tag / last character is somehow ill formated.

d-mo · 2019-09-11T17:49:25Z

@fruchtkeks @geraldpetz @kkhamruddin-pk I was able to reproduce this issue by generating and installing an SSH key that follows the new OpenSSH key format which is the default in OpenSSH version 7.8 and above. After digging deeper, I got to the point where libvirt returns the exact error as reported above by @fruchtkeks. If I try instead to add the same host using a key that follows the old PEM format then everything works fine. I'm assuming that you were all using keys that follow the new format. Can you please confirm this? The new ones begin with -----BEGIN OPENSSH PRIVATE KEY----- while the old ones begin with -----BEGIN RSA PRIVATE KEY-----.

Unfortunately the bug is within libvirt core and I'm still looking for a way to mitigate it in Mist. In the meantime, there is a simple workaround. Generate ssh keys that follow the old PEM format and use those to connect to your KVM hosts. You can generate them with the command ssh-keygen -m PEM -t rsa -b 4096.

I hope that helps.

fruchtkeks · 2019-09-13T13:08:26Z

Hi @d-mo,

thank you for the reply. I can confirm, that using the old key format fixes the issue.
Nevertheless, I did some more tests with the new key format and figured out that the issue only occurs when I copy the key directly from the terminal. It seems that the key is missing the line termination on the last line, which results in the error. When uploading the key file directly, the error does not occur.

d-mo · 2019-12-11T23:38:05Z

This has been fixed in Mist v4.2.0. Please re-open if that's not the case.

wiwa1978 mentioned this issue Oct 9, 2019

Issue KVM: Make sure ssh key and username are valid #963

Closed

d-mo closed this as completed Dec 11, 2019

d-mo mentioned this issue Feb 18, 2022

Cannot add KVM cloud. #1024

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud unavailable: Make sure ssh key and username are valid #954

Cloud unavailable: Make sure ssh key and username are valid #954

kkhamruddin-pk commented May 21, 2019

d-mo commented May 21, 2019 •

edited

Loading

kkhamruddin-pk commented May 22, 2019 •

edited

Loading

kkhamruddin-pk commented May 29, 2019

d-mo commented May 29, 2019

geraldpetz commented Jul 22, 2019

kkhamruddin-pk commented Jul 23, 2019

d-mo commented Jul 23, 2019

fruchtkeks commented Aug 22, 2019

d-mo commented Aug 22, 2019 •

edited

Loading

fruchtkeks commented Aug 22, 2019

d-mo commented Sep 11, 2019

fruchtkeks commented Sep 13, 2019

d-mo commented Dec 11, 2019

Cloud unavailable: Make sure ssh key and username are valid #954

Cloud unavailable: Make sure ssh key and username are valid #954

Comments

kkhamruddin-pk commented May 21, 2019

d-mo commented May 21, 2019 • edited Loading

kkhamruddin-pk commented May 22, 2019 • edited Loading

kkhamruddin-pk commented May 29, 2019

d-mo commented May 29, 2019

geraldpetz commented Jul 22, 2019

kkhamruddin-pk commented Jul 23, 2019

d-mo commented Jul 23, 2019

fruchtkeks commented Aug 22, 2019

d-mo commented Aug 22, 2019 • edited Loading

fruchtkeks commented Aug 22, 2019

d-mo commented Sep 11, 2019

fruchtkeks commented Sep 13, 2019

d-mo commented Dec 11, 2019

d-mo commented May 21, 2019 •

edited

Loading

kkhamruddin-pk commented May 22, 2019 •

edited

Loading

d-mo commented Aug 22, 2019 •

edited

Loading