I've configured ssh to use a different key for each server [0]. I don't bother with passphrases on each key, instead relying on a encrypted home folder. Also, I have a script [1] for rotating keys, so rotating all my credentials is a single command.

[0]: https://github.com/stevenkaras/bashfiles/blob/master/.ssh/co...

[1]: https://github.com/stevenkaras/bashfiles/blob/master/.ssh/

What good will an encrypted home folder do if your computer gets compromised? It protects against offline, physical attacks, sure, but I fail to see how it is safe against the myriad of remote attacks there exists.

It doesn't. But neither do passphrases. The key has to be in process memory at some point, and if you're running a compromised process under your user, you've already lost.

The only threat a passphrase protects against that an encrypted home doesn't is someone walking up to my unlocked laptop (it locks automatically after 5 minutes, and I make a point of locking it when I'm leaving). If you run ssh-agent, you're exposed to this threat anyways.

EDIT: After some thought, passphrases can protect against a poorly executed attack, where just files are copied wholesale, or an accidental leak. So while theoretically you're not protected, it may be sufficient for some practical security. I'll be looking into using ssh-agent in the near future.

A passphrase also protects against someone acquiring your private key by some other means. For example, if you accidentally copy the key file somewhere, or some tool (Dropbox, OwnCloud, any cloud syncing tool) syncs it to an unsecured or compromised location. The passphrase adds a knowledge factor to the key's security.

Passphrases on keys really only provide some incidental security there, for keys that you don't happen to unlock while compromised. (Or the adversary might neglect to intercept your passphrases.)

Use a yubikey or smart card to level up from these.

Something like Qubes OS is better defense against remote attacks, there is a lot of other mayhem that kind of compromise might cause. (Also passphrases don't really help there).

My ssh keypair is derived from my GPG keys. They have been generated on my yubikey neo-3. The private key has never been in any computers memory (generated on the smart card). Public keys for ssh and gpg can be downloaded at my blog.

I use ssh-agent and 1 key per computer.

If a computer and/or key is compromised, well, I just nuke all the related keys.

I don't understand the desire to manage a large number of keys since the attack surface is pretty clear:

A) The machine is not compromised and the key is safe.

B) The machine is compromised and the key should be replaced ASAP.

C) As a byproduct this forces obsolescence of keys in the ~3-4 year timeframe and you really should be swapping out keys every so often anyway. This keeps you from ignoring this fact for a decade :p

Similar, with forced key rotation. Remote key files are managed with a home brew script for assembling and distribution of authorized_keys

I use ssh-ident. It creates a separate ssh agent for each identity you use and you can setup a different identity for each host or ssh argument. And if worse comes to worse, it will prevent someone running off with all the keys you are currently using because instead they only have access to that one agent, not all your agents.

The downsides (besides possible security implications of trusting someone else's code to manage your keys) is that tools like rsync and scp won't work straight out of the box. You have to either alias ssh to ssh-ident, or provide the path to ssh-ident yourself.

https://github.com/ccontavalli/ssh-ident

Why doesn't ssh-ident modify ~/.SSH/config? Then scp (and a lot of other tools) would just work. No?

Keep it simple. I use ssh-agent .. just ~/.ssh/ - keep keys here. Backup the actual private keys and stick those on a drive you keep in a safe. Make sure your machine's HD is encrypted and you should be fine.

Is it even safe to discuss where you store your keys and how you protect them? That's halfway into a social engineering step, and sure, that's a bit of 'security by obscurity' but if I asked you where do you keep the spare keys to your house would you be comfortable answering that if you were identifiable?

If you're not comfortable answering that, perhaps it's a sign that you should be keeping them somewhere more secure.

I don't agree with that. It could be as secure as could be, but I'd still be pointing at the keys and that's information that no outsider needs to have. Why give that up? It saves them from having to look for the keys, they can now set up a targeted attack on obtaining the keys because their location is already known and that's half the battle.

Yes, not knowing where to even start the attack does make it more difficult to complete an attack. But, there is no way it could make it easier. I mean adding plastic wrap around your safe does technically make it more difficult to break into, but the issue comes where you come to rely on it. My safe is 100% uncrackable, as long as it doesn't get wet, thanks plastic wrap.

That's not the question though. The question is 'where do you keep the key to your safe'?

Hopefully it's more like "what kind of safe do you keep the keys to your other safe in?"

I think the point is in relying on something that is fundamentally secure, not secure because it is obscure. If I print my private key on a piece of paper and pin it up by my desk that's something I shouldn't tell people, but if I have it locked in an unspecified safe deposit box, there isn't much harm in describing what the deposit box is made out of.

It does make you less secure, but a good security plan starts from assuming your adversary knew all that sort of thing in the first place.

I see by the top comments that in the end the responsibility still falls on human discipline. At least two people have already confessed to having no passphrase on their SSH keys and promising that they lock their laptop screen when leaving it unsupervised. This is very insecure regardless of disk encryption.

I agree that it may decrease the security of the individual by sharing their particular security practices, but sharing good information may benefit the group.

Thanks for asking the question.

Related question: how do you manage service auth credentials for your code? E.g. client secrets for OAuth. I've never found a good solution.

The best solution I've found is git-crypt [0]. It uses a .gitattributes file and git-hooks to dynamically encrypt files whenever you push to any other user (i.e. Github). The result is that the files are plain text on your (or any other user with the password) computer but are encrypted when they leave your system.

I even leave the files encrypted during the build and deployment process and only decrypt them when they reach their final resting place. Using a combination of BASH and AWS's KMS (Kem Management Service) it wasn't too difficult and I feel much better about putting secret information in my source code. :)

[0] https://github.com/AGWA/git-crypt

Thumbs up. git-encrypt and its successor git-crypt are the best developer tools for storing password information.

We have a secrets.json in the codebase which is gitignored and we ony commit a fake secrets_sample.json.

(The actual secrets are then ridiculously insecurely transferred between colleagues via whatever communication medium the people involved think of first. Open to suggestions there :-) )

Use GPG/OpenPGP and encrypt it before sending it to your colleagues? https://gnupg.org/

Or use a zerobin instance somewhere (encrypts things clientside with AES 256) and pass them a link. https://zb.a.im/

Exactly! Just use OpenPGP! Once you've set up OpenPGP for all developers and operators that deal with secrets and production databases, it is a piece of cake to simply click the 'encrypt' button in the e-mail client whenever you need to share production log files or other potentially sensitive information with colleagues.

In an ideal world sensitive data never leaves the production environment or the machine of the developer or operator working on it, but sometimes things break down, and you need to share information to find out what went wrong, as soon as possible, with colleagues who are not necessarily physically nearby. It is best to have these tools set up and part of your work flow when you do need them.

Get everyone to sign on with Keybase.io and make GPG encryption easy (and fun!) for all. I really, really like Keybase for things like this and am trying to encourage all of my teammates to sign up, or at least use GPG encryption.

NOTE: If anyone needs an invite to Keybase.io, email me; I still have a couple left. :)

Please! I've been on the list what seems forever!

Hi, Still have any invites left?

I do - email in profile if you want one

John Resig described a way to store secret files in source control here [1]. But there is still a problem transferring encryption password securely.

[1] http://ejohn.org/blog/keeping-passwords-in-source-control/

I think the most important thing is key rotation, and generally I do it every year or so.

I prefer a single ssh key for almost everything. It's on only 1 laptop that I use daily. There is no protection on the key itself, but I always lock the laptop screen (password protected) when I leave the laptop alone.

I have other laptops/devices, usually with different keys. My "master" key is also on my 2nd laptop. Although I could have a passphrase there, I still prefer no protection except screen locking. This said, this 2nd laptop never leaves my home, where only trusted (and "innocuous") people can touch it.

A private key without a passphrase can be used by anyone who gets hold of it. Why disable the extra protection a passphrase affords? With an ssh agent you can store the passphrase in the keychain you unlock when you log on, so you won't have to type it whenever you use it, unless you ssh into your own machine and try to use the key from the remote shell (which makes sense).

Of course, in every security scenario the risks determine the level of security, but having a passphrase has no practical downsides.

I'm not arguing with you on what you said. You're certainly right. If someone gets that key, they've got access to everything.

I would add this. If someone malicious gets physical access to your laptop, you've got bigger problems to deal with. ;)

Yes, that makes sense - it would be protected at rest. (I guess the reason why I don't feel the need is that "at rest" for my specific laptop means nobody is using it and it's disconnected from the LAN/Internet)

I used to use an agent, the problem with an agent is that it will only try like 5 keys before failing. This makes it impractical to use a different key for each server.

Now I just put an identity in ~/.ssh/id_rsa and use ssh-copy-id to copy it over. Dead simple and easy. One of these days I'm going to replace the key, a script to remove ~/.ssh/authorized_keys before re-running ssh-copy-id will do the trick.

Though, these days, I'm trying to move towards making servers cattle rather than pets. I don't want to ssh into a server at all, just use configuration management to interact with them.

You can link keys to hostnames (with pattern matching too!) in ~/.ssh/config.

Thats what I do, didnt know about pattern matching though.

I use a yubikey neo + gpg smart card feature with SSH enabled gpg agent. I disable password logins and have backup key stored offline.

gpg-agent with a Yubikey holding my SSH private key.

Just the same. It's probably not perfectly secure but it beats having them on an encrypted dropbox directory which was my previous setup.

I like this solution, except for a couple of things. One, what happens if I lose my Yubikey (or it gets stolen)? Another, more realistic, problem is what do I do if I don't have my Yubikey (at my parents house)?

I have two Yubikeys, each with a separate SSH key that's been generated/signed by an offline master that resides on a USB key in a safe. In the event I lose one, I simply revoke/expire the key on keyservers and generate another using the offline key.

If you don't have your Yubikey at your parents house, maybe you should just enjoy spending time with them, instead of working on things that need an SSH key?

If you don't have your ATM-card, you can't withdraw money from your account. If you don't have your car keys, you can't use your car. This is completely desirable.

Yubikeys are very durable and small; just put them on your (physical) key-chain.

To prevent losing access when you lose your key, ensure you have a backup yubikey, or backup the keys stored on the yubikey so you can create a new one. If you use a solution where the secret cannot leave the key (such as Fido U2F keys) than register your backup key with all the services you use as well, or generate recovery codes for each service. Needless to say, these should be stored in a very secure place.

Could you expand on that a bit.

If you use an agent, just make sure you enable the option for it to prompt you upon use. This shouldn't require you to reenter your passphrase, so it's still unlocked in memory.

If you don't do this, any root user on any machine you connect to can use your ssh-agent connection to auth to other machines.

I may be mistaken, but I think that applies only if you're using agent forwarding.

Yes sorry, that's correct.

Any recommendations for backing up (private) keys? Like pgp keys for example? I consider printing them on paper, as text or QR code. Anything better than that?

Periodically copy them to an encrypted external (thumb) drive and keep that in a safe place. Use two if you want to store one of them off-site. Piece of cake with LUKS and dm-crypt on any modern Linux distro. Just use a long passphrase that you can remember.

I use Monkeysphere (http://web.monkeysphere.info/), which aims to use OpenPGP as a PKI for SSH keys. It basically provides scripts to generate the authorized_keys file from a list of a OpenPGP key IDs (on the server side) and feed the GPG private key material to ssh-agent (on the client side). I think the client step can also be used directly by gpg-agent (which can play the part of the ssh-agent as well), but I am not using it. I am overall quite satisfied, but see below.

PROS:

* When you update your trusted GPG certificates (adding new auth keys, revoking others) the authorized_keys files get updated at the first execution of monkeysphere on the server (you usually put that in cron together with gpg --refresh-keys). So you can rekey without having to change manually all the SSH accounts you have.

* You do not need to recompile or patch SSH and it is compatible with other keys not fed by Monkeysphere.

* The GPG PKI, although not perfect, has quite some features; in particular, it lets you somewhat easily manage different keys on different computers, generate and revoke subkeys independently. The web of trust also helps you when trusting keys from other people.

* Monkeysphere can also be used for SSL certificates, although that is more difficult and less supported (and also less useful, now that we have Let's Encrypt).

CONS:

* Monkeysphere's development appears to be a bit stalled; not the ideal situation for a security-related thing.

* Monkeysphere does a good job, but it should not be trusted blindly. There may be a number of situation where external conditions may break the game; e.g, if you do not realize there is a misconfiguration, a revoked key may remain in authorized_keys because Monkeysphere is failing at updating; if you trusted Monkeysphere to do everything correctly, you would be exposed without knowing it.

* The GPG PKI as well is not perfect; for example, key management is complicated with many subkeys (for instance, you cannot give meaningful names to them) and the web of trust mechanism does not support "scoped trust" (i.e., giving different trust levels for different things you want to do).

* Monkeysphere only works when the remote host is a Linux box where you can install Monkeysphere and have it update authorized_keys via cron. No hope to manage GitHub keys or things like that (unless they introduce support, which seems unlikely).

ssh-agent with IdentitiesOnly. Fairly regular key rotation schedule, with separate keys for groups of servers based on a loose taxonomy.