The goal of this document is to help operational teams with the configuration of OpenSSH server and client.
All Mozilla sites and deployment should follow the recommendations below.
The Security Assurance and Security Operations teams maintain this document as a reference guide.
Only non-default settings are listed in this document
Most default OpenSSH settings that are security-related already provide good security, thus changing them is at your own risk and is not documented here. For example, these guidelines assume only SSH protocol 2 is configured in the server, and SSH protocol 1 is disabled. This also assumes that you are keeping OpenSSH up-to-date with security patches.
See man sshd_config, man ssh_config for more information on specific settings if you nevertheless need to change them. |
OpenSSH server
Configuration
Different versions of OpenSSH support different options which are not always compatible. This guide shows settings for the most commonly deployed OpenSSH versions at Mozilla - however, using the latest version of OpenSSH is recommended.
Modern (OpenSSH 6.7+)
File: /etc/ssh/sshd_config
# Supported HostKey algorithms by order of preference.HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
# Password based logins are disabled - only public key based logins are allowed.AuthenticationMethods publickey
# LogLevel VERBOSE logs user's key fingerprint on login. Needed to have a clear audit track of which key was using to log in.LogLevel VERBOSE
# Log sftp level file access (read/write/etc.) that would not be easily logged otherwise.Subsystem sftp /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO
# Root login is not allowed for auditing reasons. This is because it's difficult to track which process belongs to which root user:## On Linux, user sessions are tracking using a kernel-side session id, however, this session id is not recorded by OpenSSH.# Additionally, only tools such as systemd and auditd record the process session id.# On other OSes, the user session id is not necessarily recorded at all kernel-side.# Using regular users in combination with /bin/su or /usr/bin/sudo ensure a clear audit track.PermitRootLogin No
File: /etc/ssh/moduli
All Diffie-Hellman moduli in use should be at least 3072-bit-long (they are used for diffie-hellman-group-exchange-sha256) as per our Key management Guidelines recommendations. See also man moduli.
To deactivate short moduli in two commands: awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.tmp && mv /etc/ssh/moduli.tmp /etc/ssh/moduli
Intermediate (OpenSSH 5.3)
This is mainly for use by RHEL6, CentOS6, etc. which run older versions of OpenSSH.
File: /etc/ssh/sshd_config
# Supported HostKey algorithms by order of preference.HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512,hmac-sha2-256
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
# Password based logins are disabled - only public key based logins are allowed.RequiredAuthentications2 publickey
# RequiredAuthentications2 not work on official OpenSSH 5.3 portable.# In this is your case, use this instead:#PubkeyAuthentication yes#PasswordAuthentication no# LogLevel VERBOSE logs user's key fingerprint on login. Needed to have a clear audit track of which key was using to log in.LogLevel VERBOSE
# Log sftp level file access (read/write/etc.) that would not be easily logged otherwise.Subsystem sftp /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO
# Root login is not allowed for auditing reasons. This is because it's difficult to track which process belongs to which root user:## On Linux, user sessions are tracking using a kernel-side session id, however, this session id is not recorded by OpenSSH.# Additionally, only tools such as systemd and auditd record the process session id.# On other OSes, the user session id is not necessarily recorded at all kernel-side.# Using regular users in combination with /bin/su or /usr/bin/sudo ensure a clear audit track.PermitRootLogin No
File: /etc/ssh/moduli
All Diffie-Hellman moduli in use should be at least 2048-bit-long. From the structure of moduli files, this means the fifth field of all lines in this file should be greater than or equal to 2047.
To deactivate weak moduli in two commands: awk '$5 >= 2047' /etc/ssh/moduli > /etc/ssh/moduli.tmp; mv /etc/ssh/moduli.tmp /etc/ssh/moduli
Multi-Factor Authentication (OpenSSH 6.3+)
Recent versions of OpenSSH support MFA (Multi-Factor Authentication). Using MFA is recommended where possible.
ATTENTION In order to allow using one time passwords (OTPs) and any other text input, Keyboard-interactive is enabled in OpenSSH. This MAY allow for password authentication to work. It is therefore very important to check your PAM configuration so that PAM disallow password authentication for OpenSSH.
OpenSSH 6.3+ (default)
File: /etc/ssh/sshd_config
# IMPORTANT: you will have to ensure OpenSSH cannot authenticate with passwords with PAM in /etc/pam.d/sshd# "PasswordAuthentication no" is not sufficient!PubkeyAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive:pam
KbdInteractiveAuthentication yes
UsePAM yes
# Ensure /bin/login is not used so that it cannot bypass PAM settings for sshd.# Note, this option is no longer needed as of OpenSSH 7.4 as support for UseLogin has been removedUseLogin no
OpenSSH 5.3+ w/ RedHat/CentOS patch (old)
File: /etc/ssh/sshd_config
# Allow keyboard-interactive.# IMPORTANT: you will have to ensure OpenSSH cannot authenticate with passwords with PAM in /etc/pam.d/sshd# "PasswordAuthentication no" is not sufficient!RequiredAuthentications2 publickey,keyboard-interactive:skey
PasswordAuthentication no
ChallengeResponseAuthentication yes
UsePAM yes
# Ensure /bin/login is not used so that it cannot bypass PAM settings for sshd.UseLogin no
PAM configuration for use with the OATH Toolkit or DuoSecurity as second authentication factor.
File: /etc/pam.d/sshd
#%PAM-1.0auth required pam_sepermit.so
# WARNING: make sure any password authentication module is disabled.# Example: pam_unix.so, or "password-auth", "system-auth", etc.#auth include password-auth# Options to enable when using OATH toolkit#auth requisite pam_oath.so usersfile=/etc/users.oath digits=6 window=20# Options to enable when using DuoSecurity#auth sufficient /lib64/security/pam_duo.soaccount required pam_nologin.so
The PAM stack in this scenario executes the following logic (in our example we
follow the flow with pam_duo.so in use)
The pam_sepermit.so module is called which checks if the user attempting to
log in via SSH is present in the /etc/security/sepermit.conf.
If the user is present in the config file, and the config asserts that the user
can only log in if SELinux is enforcing, and SELinux is not enforcing, then
the PAM control of required prevents the user from logging in (though PAM
would continue down the stack).
The password-auth include is commented out and skipped
The /lib64/security/pam_duo.so module is called and the user is prompted for
a duo MFA code.
If the code provided is correct PAM immediately permits the user access and
doesn’t continue executing.
If the code provided is incorrect, PAM continues down the stack
The pam_nologin.so checks if the file /etc/nologin exists and if so blocks
access to the user.
If at the end of the stack, the single sufficient control of pam_duo.so
did not return a success, PAM defaults to deny and denies the login.
Ciphers and algorithms choice
When CHACHA20 (OpenSSH 6.5+) is not available, AES-GCM (OpenSSH 6.1+) and any other algorithm using EtM (Encrypt then MAC) disclose the packet length - giving some information to the attacker. Only recent OpenSSH servers and client support CHACHA20.
NIST curves (ecdh-sha2-nistp512,ecdh-sha2-nistp384,ecdh-sha2-nistp256) are listed for compatibility, but the use of curve25519 is generally preferred.
# Hardened SSHD bastion config -- modify wisely!# Based on https://wiki.mozilla.org/Security/Guidelines/OpenSSH# With additional restrictions where applicable# -lo and -rt users only have local console loginDenyUsers *-rt
DenyUsers *-lo
# hardened params follow. every non-needed feature is disabled by default,# following the principle of least rights and least features (more enabled# features mean a more important attack surface).# === FEATURES ===# disable non-needed sshd featuresAllowAgentForwarding no
AllowTcpForwarding no
AllowStreamLocalForwarding no
X11Forwarding no
PermitTunnel no
PermitUserEnvironment no
PermitUserRC no
GatewayPorts no
# === INFORMATION DISCLOSURE ===# don't yell to the world that we're running debian,# this disables the debian string version on the server hello messageDebianBanner no
# however, display a legal notice for each connectionBanner /etc/ssh/banner
# don't print the bastion MOTD on connectionPrintMotd no
# === CRYPTOGRAPHY ===# enforce the use of ssh version 2 protocol, version 1 is disabled.# all sshd_config options regarding protocol 1 are therefore omitted.Protocol 2# only use hostkeys with secure algorithms, and omit the ones using NIST curvesHostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
# list of allowed ciphers.# chacha20-poly1305 is a modern cipher, considered very secure# aes is still the standard, we prefer gcm cipher mode, but also# allow ctr cipher mode for compatibility (ctr is still considered secure)# we deny arcfour(rc4), 3des, blowfish and castCiphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
# list of allowed message authentication code algorithms.# etm (encrypt-then-mac) are considered the more secure, we# prefer umac (has been proven secure) then sha2.# for older ssh client, fallback to the non-etm version of# the algorithms.# we deny md5 and sha1MACs umac-128-etm@openssh.com,umac-64-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128@openssh.com,umac-64@openssh.com,hmac-sha2-512,hmac-sha2-256
# List of allowed key exchange algorithms.# we prefer curve25519-sha256 which is considered the most modern/secure,# and still allow diffie hellman with group exchange using sha256 which is# the most secure dh-based kex.# we avoid algorithms based on the disputed NIST curves, and anything based# on sha1.KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
# force rekey every 512M of data or 6 hours of connection, whichever comes firstRekeyLimit 512M 6h
# === AUTHENTICATION ===# we allow only public key authentication ...PubkeyAuthentication yes
# ... not passwordPasswordAuthentication no
# ... keyboard interactive (needed for MFA through PAM)KbdInteractiveAuthentication yes
# ... not kerberosKerberosAuthentication no
# ... challenge-response (needed for MFA through PAM)ChallengeResponseAuthentication yes
# ... not host-basedHostbasedAuthentication no
# ... and not gssapi auth.GSSAPIAuthentication no
GSSAPIKeyExchange no
# just in case, we also explicitly deny empty passwordsPermitEmptyPasswords no
# this needs to be set at "yes" to allow PAM keyboard-interactive authentication,# which is not a security issue because the AuthenticationMethods below force the use of# either publickey or publickey+keyboard-interactive, hence password-only login is never# possible, for root or any other account for that matterPermitRootLogin yes
# === LOGIN ===# disconnect after 30 seconds if user didn't log in successfullyLoginGraceTime 30# not more than 1 session per network connection (connection sharing with ssh client's master/shared mode)MaxSessions 10# maximum concurrent unauth connections to the sshd daemonMaxStartups 50:30:500
# accept LANG and LC_* vars (also includes LC_BASTION)AcceptEnv LANG LC_*
# === SYSTEM ===# sshd log level at verbose in auth facility for auditing purposesLogLevel VERBOSE
SyslogFacility AUTH
# check sanity of user HOME dir before allowing user to loginStrictModes yes
# never use dns (slows down connections)UseDNS no
# use PAM facilityUsePAM yes
# === AuthenticationMethods vs potential root OTP vs potential user MFA ===# 2FA has been configured for root, so we force pubkey+PAM for it#Match User root# AuthenticationMethods publickey,keyboard-interactive:pam# Unconditionally skip PAM auth for members of the bastion-nopam groupMatch Group bastion-nopam
AuthenticationMethods publickey
# if in one of the mfa groups, use pamMatch Group mfa-totp-configd
AuthenticationMethods publickey,keyboard-interactive:pam
Match Group mfa-password-configd
AuthenticationMethods publickey,keyboard-interactive:pam
# by default, always ask the publickey (no PAM)Match All
AuthenticationMethods publickey
Subsystem sftp /usr/libexec/sftp-server
OpenSSH client
Configuration
If you have a file containing known_hosts using RSA or ECDSA host key algorithm and the server now supports ed25519 for example, you will get a warning that the host key has changed and will be unable to connect. This means you will have to verify the new host key.
The following configurations expect a recent OpenSSH client, as updating OpenSSH on the client side is generally not an issue.
Modern
This configuration is less compatible and you may not be able to connect to some servers which use insecure, deprecated algorithms. Nevertheless, modern servers will work just fine.
File: ~/.ssh/config
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSHHostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
Intermediate (connects to older servers)
This configuration can connect to older OpenSSH servers which run old or intermediate configurations.
File: ~/.ssh/config
# Ensure KnownHosts are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.HashKnownHosts yes
# Host keys the client accepts - order here is honored by OpenSSHHostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
Key generation
Large key sizes are used as SSH keys are not renewed very often (see also Key management Guidelines).
Don’t hesitate to create multiple different keys for different usages. In particular, never mix your personal and Mozilla keys.
# RSA keys are favored over ECDSA keys when backward compatibility ''is required'',# thus, newly generated keys are always either ED25519 or RSA (NOT ECDSA or DSA).$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_mozilla_$(date +%Y-%m-%d) -C "Mozilla key for xyz"# ED25519 keys are favored over RSA keys when backward compatibility ''is not required''.# This is only compatible with OpenSSH 6.5+ and fixed-size (256 bytes).$ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_mozilla_$(date +%Y-%m-%d) -C "Mozilla key for xyz"
You may then want to add the new key to your SSH agent or your configuration file (or both).
# Add key to ssh-agent$ ssh-add ~/.ssh/id_..._mozilla... # <= replace by your key's path# Add configuration to ~/.ssh/confighost *.mozilla.com
IdentityFile ~/.ssh/id_...mozilla... # <= replace by your key's path
Protection of user keys
Protected by strong passphrase.
Never copied to another system than your own workstation/personal physical disks/tokens.
Use SSH forwarding or SSH tunneling if you need to jump between hosts. DO NOT maintain unnecessary agent forwarding when unused.
Protection of machine keys
When SSH keys are necessary for automation between systems, it is reasonable to use passphrase-less keys.
The recommended settings are identical to the user keys.
The keys must be accessible only by the admin user (root) and/or the system user requiring access.
Usage of machine keys should be registered in an inventory (a wiki page, ldap, an inventory database), to allow for rapid auditing of key usage across an infrastructure.
The machine keys should be unique per usage. Each new usage (different service, different script called, etc.) should use a new, different key.
Only used when strictly necessary.
Restrict privileges of the account (i.e. no root or “sudoer” machine account).
Using a ForceCommand returning only the needed results, or only allowing the machine to perform SSH-related tasks such as tunneling is preferred.
Disable sftp if not needed as it exposes more surface and different logging mechanisms than SSH (and thus scp) itself.
Machine keys do not play well with multi-factor authentication as there is no human interaction.
All logins from machine accounts should be protected by an additional authentication layer (VPN, another machine, etc.).
All logins from machine accounts are only allowed within the private IP-space, and if possible, only the relevant machine source IPs should be accessible.
File: /etc/ssh/sshd_config (OpenSSH 6.3+)
Match User machine_user Address 10.0.0.0/8,192.168.0.0/16,172.16.0.0/12
PubkeyAuthentication yes
KbdInteractiveAuthentication no
AuthenticationMethods publickey
Match User machine_user Address 10.0.0.0/8,192.168.0.0/16,172.16.0.0/12
RequiredAuthentications2 publickey
Auditing your existing SSH keys
Existing keys are generally stored in ~/.ssh/ (Linux/OSX) or %APPDATA% (Windows). Look for id_{rsa,ed25519,ecdsa,dsa}, identity, IdentityFile, *.pem, and other identity files.
Display SSH keys information
# You may run this for any key file that you have.$ ssh-keygen -lf id_rsa
2048 bc:4f:46:2b:3d:f1:e2:0f:ac:40:99:49:ed:c9:81:a2 Mozilla key for xyz (RSA)^^ ^^^^^^^^^ ^^^^ ^^^
|__ Size |__ Fingerprint |__ Comment |__ Type
SSH agent forwarding
ATTENTION SSH Agent forwarding exposes your authentication to the server you’re connecting to. By default, an attacker with control of the server (i.e. root access) can communicate with your agent and use your key to authenticate to other servers without any notification (i.e. impersonate you).
For this reason, one must be careful when using SSH agent forwarding. Defaulting to always forwarding the agent is strongly discouraged.
Note also that while the attacker can use your key as long as the agent is running and forwarded, he cannot steal/download the key for offline/later use.
SSH forwarding allows you to jump between hosts while keeping your private key on your local computer. This is accomplished by telling SSH to forward the authentication requests back to the ssh-agent of your local computer. SSH forwarding works between as many hosts as needed, each host forwarding new authentication request to the previous host, until the ssh-agent that holds the private key is reached.
On each host, two environment variables are declared for the user enabling ssh-agent:
$SSH_AUTH_SOCK declares the location of the unix socket that can be used to forward an authentication request back to the previous host.(ex: /tmp/ssh-NjPxtt8779/agent.8779). Only present if using SSH agent forwarding.
$SSH_CONNECTION shows the source IP and port of the previous host, as well as the local IP and port. (ex: 10.22.248.74 44727 10.8.75.110 22).
To use ssh-agent, add the flag -A to your ssh commands:
$ ssh -A user@ssh.mozilla.com
You can set the following configuration parameter in your local ssh configuration at ~/.ssh/config.
Host ssh.mozilla.com
ForwardAgent yes
Hardening the Agent forwarder
It is possible to require confirmation every time the agent is used (i.e. when you connect to a server through the SSH agent) by using the -c flag:
# First, remove the key from the agent if it's already loaded:
$ ssh-add -d ~/.ssh/id_ed25519
# And re-add it with the -c flag:
$ ssh-add -c ~/.ssh/id_ed25519
It is also possible to lock the key in the agent after a configurable amount of time, this can be done either for all keys when starting the agent, or per key when adding the keys to the agent with the -t flag:
# Keep all keys decrypted/useable in memory for 30 minutes (1800 seconds)
$ ssh-agent -t 1800
# First, remove the key from the agent if it's already loaded:
$ ssh-add -d ~/.ssh/id_ed25519
# Re-add it, with the -t flag to keep this specific key decrypted/useable in memory for 30 minutes (1800 seconds)
$ ssh-add -t 1800 ~/.ssh/id_ed25519
For MacOSX in particular it’s possible to save the passphrase in the Keychain. If you do so it is strongly recommended to also change the keychain setting to lock itself when the computer is locked, and/or to timeout and lock the keychain. These settings are not controlled by OpenSSH.
# MacOSX only - save the passphrase in the Keychain
$ ssh-add -K ~/.ssh/id_ed25519
Recommended, safer alternatives to SSH agent forwarding
OpenSSH >=7.3
OpenSSH 7.3 onwards allow users to jump through several hosts in a rather automated fashion. It has full support for scp and sftp commands as well as regular ssh.
For example to reach a host behind a bastion/jumphost:
# Single jump
$ ssh -J ssh.mozilla.com myhost.private.scl3.mozilla.com
# Jump through 2 hops
$ ssh -J ssh.mozilla.com,ec2-instance.aws.net 10.0.0.3
# Copy data from a host
$ scp -oProxyJump=ssh.mozilla.com myhost.private.scl3.mozilla.com:/home/jdoe/testfile ./
You can also add these lines to your ~/.ssh/config
It is possible to directly forward ports for single jumps instead of forwarding the agent. This has the advantage of never exposing your agent to the servers you’re connecting to.
For example, you can add these lines to your ~/.ssh/config
This will automatically forward the SSH connection over ssh.mozilla.com when you connect to a mozilla.com SSH server.
Appendixes
Key material handling
Key material identifies the cryptographic secrets that compose a key. All key material must be treated as MOZILLA CONFIDENTIAL GROUP RESTRICTED data, meaning that:
Only individual with specific training and need-to-know should have access to key material.
Key material must be encrypted on transmission.
Key material can be stored in clear text, but only with proper access control (limited access).
This includes:
OpenSSH server keys (/etc/ssh/ssh_host_*key)
Client keys (~/.ssh/id_{rsa,dsa,ecdsa,ed25519} and ~/.ssh/identity or other client key files).
Client key size and login latency
In order to figure out the impact on performance of using larger keys - such as RSA 4096 bytes keys - on the client side, we have run a few tests:
On an idle, i7 4500 intel CPU using OpenSSH_6.7p1, OpenSSL 1.0.1l and ed25519 server keys the following command is ran 10 times:
time ssh localhost -i .ssh/id_thekey exit
Results:
Client key
Minimum
Maximum
Average
RSA 4096
120ms
145ms
127ms
RSA 2048
120ms
129ms
127ms
ed25519
117ms
138ms
120ms
Keep in mind that these numbers may differ on a slower machine, and that this contains the complete login sequence and therefore is subject to variations. However, it seems safe to say that the latency differences are not significant and do not impact performance sufficiently to cause any concern regardless of the type of key used.