Over the weekend, Liz Fong-Jones, a Developer Advocate at honeycomb.io posted her experience with the security hardening of honeycomb.io’s infrastructure. In her post, on GitHub, Liz explains how SSH keys, which provide authentication between hosts, can be vulnerable to different threats, which might be overlooked.
Liz mentions that by adding passphrase encryption, the private keys become resistant to theft when at rest. However, when they are in use, the usability challenges of re-entering the passphrase on every connection means that “engineers began caching keys unencrypted in memory of their workstations, and worse yet, forwarding the agent to allow remote hosts to use the cached keys without further confirmation”.
The Matrix breach, which took place on April 11 showcases an example of what happens when authenticated sessions are allowed to propagate without a middle-man. The intruder in the Matrix breach had access to the production databases, potentially giving them access to unencrypted message data, password hashes, and access tokens.
Liz also mentions two primary ways of preventing an attacker from misusing credentials.
- Using a separate device that generates, using a shared secret, numerical codes that we can transfer over out of the band and enter alongside the key.
- Having a separate device perform all the cryptography only when physically authorized by the user.
In her post, Liz asks, “What will work for a majority of developers who are used to simply loading their SSH key into the agent at the start of their login session and SSHing everywhere?” and also shares her work on how one can avoid such threats.
Some pre-requisites to this that Liz mentions is, “I’m assuming that you have a publicly exposed bastion host for each environment that intermediates accesses to the rest of each environment’s VPC, and use SSH keys to authenticate from laptops to the bastion and from the bastion to each VM/container in the VPC”.
As a preliminary step, the user should start by enabling numerical time-based one-time password (TOTP) for SSH authentication. However, since a malicious host could impersonate the real bastion (if strict host checking isn’t on), intercept the OTP, and then use it to authenticate to the real bastion, “ it’s better than being wormed or compromised because you forgot to take basic measures against even a passive adversary”, Liz states.
After the server and the client setup, the user needs to use Chef to populate /etc/2fa_token_keys with keys that are generated and stored securely. There are different setup methods including:
Mac client setup
Users with Touchbar Macs should use TouchID to authenticate logins, as they’ll have their laptop and their fingers with them anyways. For instance, SeKey is an SSH Agent that allows users to authenticate to UNIX/Linux SSH servers using the Secure Enclave.
Krypt.co setup for iOS and Android
With the help of krypt.co, instead of generating OTPs and sending them over manually, the mobile devices can securely store our SSH keys and only remotely authorize usage (and send the signed challenge to the remote server) simply with a single click.
This process is even more secure than a TOTP app so long as the user supplies appropriate parameters to force hardware coprocessor storage (NIST P-256 for iOS, and 3072-bit RSA for Android, on new enough devices). Make sure people use screen locks!
Liz in her post also explores YubiKey hardware token & Linux/ChromeOS client setup. To know more about this and how to set up in detail, read Liz’s GitHub post.