Can’t SSH? You Might Have Too Many Keys
Welcome to another episode of Tom Discovers The Quirks of Linux, where we spend an extraordinary amount of time doing basic things 😩
In today’s episode:
Why is my SSH key getting rejected? It worked a few days ago! (WTF Edition)
So, my SSH key no longer works
I had a problem this week where I couldn’t SSH onto a server.
I swore to myself that it had worked a few days ago.
On Monday, I created a new SSH key pair on my laptop, added the public key into the authorized_keys on the remote server, and I was able to authenticate with that key.
But then by Thursday, the key seemed to stop working.
I was using the ssh command to log on, passing an identity file, like this:
ssh -i my-key.pem ec2-user@some-aws-host.puppies
Worked on Monday. Stopped working on Thursday.
What’s the issue?
Let’s investigate
I started investigating in the only way I knew how. Getting some logs.
Put some logs on the fire
I turned on the verbose output from ssh, by adding the -v switch:
ssh -v -i my-key.pem ec2-user@some-aws-host.puppies
When I ran this command, I saw that the ssh program was trying multiple keys, one after another, even though I was only specifying one identity file:
debug1: Next authentication method: publickey
debug1: Offering public key: <REDACTED> agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: <REDACTED> agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: <REDACTED> agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: <REDACTED> agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: <REDACTED> agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: <REDACTED> agent
Received disconnect from 1.2.3.4 port 22:2: Too many authentication failures
Why was ssh trying to use multiple keys?
Also, why was my identity file not even in that list?
I’ve redacted the keys above, but the file I was specifying with -i my-key.pem didn’t even show up in the logs.
Clearly ssh wasn’t even trying the key at all! Rude.
Has anything changed recently?
I thought about the changes which I’d made to my laptop or the server, in the last few days, which might have caused this to happen.
I realised there was one change which might have affected things:
I had created a new SSH key, for another project I was working on.
But how would creating a new key affect this SSH connection?
Bizarre ssh love triangle
Welcome to the ssh love triangle. There are three characters in this particular romance:
-
ssh-agent - the program that holds your authentication keys (identities) and shares them with ssh
-
ssh - the remote login client that you use when typing
ssh -
ssh_config - the config file you can use to provide options to the ssh client
Let’s find out how these play together. LEARNING IS FUN
First stop, ssh-agent. This holds your keys for you. In the manpages for ssh-agent (man ssh-agent), it says this:
Multiple identities may be stored in ssh-agent concurrently and ssh(1) will automatically use them if present.
This means: when ssh asks ssh-agent for identities, ssh might receive multiple identities, and it will try to use all of them.
So how do you know which identities the agent has? Curiously, the command to list all identities is ssh-add -L.
I ran this command and it said that I have 6 keys (it seems that I like collecting keys 🔑🔑):
$ ssh-add -L
ssh-rsa <REDACTED> ...
ssh-rsa <REDACTED> ...
ssh-rsa <REDACTED> ...
ssh-rsa <REDACTED> ...
ssh-rsa <REDACTED> ...
ssh-rsa <REDACTED> ...
Now we know that ssh-agent will offer 6 keys.
How does ssh use these keys/identities? This time, let’s check man ssh_config:
any identities represented by the authentication agent will be used for authentication unless IdentitiesOnly is set.
What this is really saying, is that ssh will greedily try every identity that it’s given, unless you tell it otherwise, using this IdentitiesOnly setting.
Can you have too many keys?
But you’re probably thinking: You gave the correct key with -i, so why was ssh not able to authenticate with that key?
This is because of the number of different authentication attempts that are allowed by the ssh daemon on the remote server.
The server-side ssh daemon config is stored in /etc/ssh/sshd_config (this is an Amazon Linux server - other Linuxes may vary!). It contains this line:
MaxAuthTries 6
This means that the ssh daemon on the server will only allow 6 attempts at authentication before it will fail and close the connection.
This is to stop you from opening an ssh connection and trying a million different keys.
So it seems that my ssh client tried to authenticate with 6 different keys, got rejected, and didn’t even have a chance to give the correct key.
And now, I think we’re ready for a solution.
Towards a solution
Let’s recap. We now know that:
-
ssh-agent can manage many keys. Mine has 6.
-
ssh will try all of these keys, plus any extra identity files that it’s given.
-
The server only allows a maximum of 6 authentication attempts.
The solution is to force ssh to use only the identity file I’ve given with -i.
We do that with the setting IdentitiesOnly yes. We can give the setting as an option at the command-line, like this:
ssh -o 'IdentitiesOnly yes' -i keypair.pem ec2-user@some-amazon-host
Now, ssh will use only the keypair I’ve just given.
I can switch on verbose output (-v) to confirm this:
debug1: Will attempt key: keypair.pem explicit
debug1: Trying private key: keypair.pem
...
debug1: Authentication succeeded (publickey).
And finally, we get into the server:
$ ssh -v -o 'IdentitiesOnly true' -i keypair.pem ec2-user@some-aws-host.puppies
...
Last login: Fri Mar 19 08:42:05 2021 from 1.2.3.4
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-4-3-2-1 ~]$
Nothing’s ever easy, eh?