Re: Clusters, known_hosts, host keys, and "REMOTE HOST IDENTIFICATION HAS CHANGED"

Richard Conto <richard.conto@xxxxxxxxx> · Fri, 18 Sep 2009 14:02:58 -0400

I suspect the problem is that the application specific IP addresses  

move from host/server to server, and since the Host Key is tied to the  

host/server rather than IP address - the SSH clients get worried.

If you have control over your user's SSH clients, then you could  

implement something like #4 below.  If you don't have control over  

your user's SSH clients, you're hosed and have to run separate  

instances of sshd bound individually to each IP address - the  

management headache you're trying to avoid.

I would suggest as an enhancement to #4 that the SSH client be allowed  

to define a named pool (of the server hosts) and a list of  

"swimmers" (awful name) that would accept any of the host keys for the  

pool.  That way, you could have distinct, multiple pools.  Each pool  

could be administered separately, and a "swimmer" could only be in a  

single pool at a time.

I don't see a good way around this that isn't a management nightmare.   

Either you're managing sshd bound to an IP address rather than a host,  

or you're managing some client configuration.  At least, by managing  

the sshd's, you're not binding yourself to any particular version of  

the ssh client and can take advantage of any bug fixes or other  

enhancements that might occur in the client.

On Sep 18, 2009, at 1:08 PM, H. Kurth Bemis wrote:

On Thu, 2009-09-17 at 16:53 -0700, Steve Bonds wrote:

SSH List-dwellers:

I'm using OpenSSH in an environment with lots of clusters.  These
clusters have IP addresses which are associated with a particular
application rather than with a particular host.  Oftentimes
(especially for file transfers) it's helpful to ssh/scp to the IP

address associated with the application rather than the one  

associated

with the host.  However, given that each host has its own host key,  

we

frequently get:

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

Which of course panics the user the first time they see it, and  

causes

them to ignore it the second time onward-- neither of which are
desired behaviors...

I've thought about several solutions to this including:

1) Make all the host keys the same (hundreds of hosts, kind of
diminishes the value of a host key...)
2) Configure ssh to ignore host key changes (harder than you might
think since often new ssh clients are brought in)

3) Give each application its own dedicated ssh and host key (tricky  

to

set up and monitor, fairly high effort)
4) Tweak OpenSSH so that it will accept any host key from a list
(requires some programming effort, might not be a good idea)
5) Other?

What do you all think of option 4?  In particular, I was thinking  

that

there might be a way to allow hosts on the same subnet to simply
prompt to add the additional key for the same DNS name rather than

popping up the man-in-the-middle warning.  If there were multiple  

keys

present in known_hosts for a given hostname, any of them would be
accepted.

Could this be done without weakening the host security of OpenSSH?
Should I instead just hold The Great Re-Keying and go with option 1?

I appreciate any advice.

Thanks,

 -- Steve Bonds

Maybe the issue doesn't really involve modifying OpenSSH at all.  If  

you

have access to the hosts, wouldn't it be possible to

pre-generate .known_hosts with all the host keys in your cluster?   

Then

each client would have every key in it's .known_hosts, so it wouldn't
matter which host the client was connecting to.

Then if one of the keys change, you can generate a new .known_hosts.
Users are still alerted if a key changes on it's own.

Whatever your final solution, please remember to share with the
class. :]

~k