552 lines
18 KiB
Markdown
552 lines
18 KiB
Markdown
---
|
||
title: Hardening Linux - An Update to Securing SSH
|
||
description: "A Guide to Making Linux a Little more secure"
|
||
tags: ["Linux", "Ubuntu"]
|
||
author: "ahmza"
|
||
date: 2025-06-05T15:34:10+01:00
|
||
draft: true
|
||
weight:
|
||
featureimage:
|
||
lastmod:
|
||
---
|
||
|
||
Update to [Securing SSH](/posts/securing-ssh/)
|
||
|
||
{{< alert icon="fire" >}}
|
||
|
||
> - This Guide assumes a Linux -> Linux SSH Session
|
||
> - Please take care with production systems
|
||
> - I take no responsibility for misconfigured systems and lockouts
|
||
> - Try in a VM first and thank me later :)
|
||
> - My Commands will be for Debian 13 Trixie
|
||
|
||
{{< /alert >}}
|
||
|
||
## Introduction
|
||
|
||
This guide will go over the basic steps of hardening a Linux machine with a
|
||
focus on remote access. The reason this is important is because when we begin to
|
||
expose services to the internet and make them publicly accessible then that means
|
||
anyone can probe around.
|
||
|
||
If you have not heard of _Overlay Networks_ before they're similar to
|
||
VPNs however do not need any open ports as they use NAT Punching.
|
||
|
||
Check out my post on [Netbird](/posts/no-more-port-forwarding/#netbird) an alternative to [Tailscale](/posts/no-more-port-forwarding/#tailscale)
|
||
|
||
Follow along and get your remote systems secure !!
|
||
|
||
## Initial Steps
|
||
|
||
Before we get into the meat and potatoes of this guide, it would be a good idea
|
||
to go over the following steps, especially if you are configuring a new install.
|
||
|
||
### Software Updates
|
||
|
||
Software updates are key for security as they will often include security
|
||
patches. Depending on your flavour of Linux this command will vary.
|
||
|
||
```bash
|
||
sudo apt update && sudo apt upgrade
|
||
```
|
||
|
||
{{< alert "bell" >}}
|
||
If you do not expect to be logging into this remote machine very often
|
||
then using [Ansible to update Remote Machines](/posts/ansible-to-manage-remote-servers)
|
||
is a great alternative to the manual approach
|
||
{{< /alert >}}
|
||
|
||
### User Accounts
|
||
|
||
It's not a good idea to continue using the root account after install, reason being
|
||
the root account has complete control over the system, whereas a user account is more
|
||
restricted by default. Logging is more granular with different user accounts,
|
||
especially with more than one user.
|
||
|
||
The following command will allow you to create a new user, it will also ask you
|
||
some questions about user information however the only required fields are the
|
||
username and password.
|
||
|
||
```bash
|
||
adduser username
|
||
```
|
||
|
||
now we can add our newly created user to the sudo group to give admin permissions
|
||
|
||
```bash
|
||
usermod -aG sudo username
|
||
```
|
||
|
||
{{< alert "bell" >}}
|
||
Remember to replace username with your desired username
|
||
{{< /alert >}}
|
||
|
||
you can now logout of root and log back in as your user. Further action on the
|
||
server should now be executed from this user account.
|
||
|
||
## Creating SSH Keys
|
||
|
||
Now that you have your own user account, we can create an SSH key for it. An SSH
|
||
Key is a Public and Private key pair which allow client and server to
|
||
authenticate using Public Key cryptography. Public Key cryptography is much
|
||
harder to crack compared to a password, also we no longer send passwords across
|
||
networks which can be at risk of interception.
|
||
|
||
SSH can also be more convenient once you set up ssh-agent which will be covered
|
||
later in the guide, with password authentication you need to enter a password
|
||
everytime, however with ssh-agent you can enter your ssh-key password once and
|
||
it remains unlocked for the remainder of the session.
|
||
|
||
{{< alert "fire" >}}
|
||
**IMPORTANT:** **NEVER** share your private ssh key ... it's private for a reason.
|
||
{{< /alert >}}
|
||
|
||
By running the following command you will create an ssh-key !!
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -C NameYourKey
|
||
```
|
||
|
||
- `-t` - specifies the type of cryptographic algorithm we will use
|
||
- `-C` - notice it's capital, A name for your ssh keys. I usually give them the
|
||
hostname of the local machine
|
||
|
||
{{< alert "bell" >}}
|
||
If you would like a deep dive into why I pick this algorithm, click the drop down, else skip ahead :D
|
||
{{< /alert >}}
|
||
{{< details >}}
|
||
The `-t` flag specifies the type of encryption to use. The reasons I have gone with `ed25519`:
|
||
|
||
- `ed25519` is faster and generation, signing and verification operations
|
||
- Believed to be more resistant to certain attacks
|
||
- shorter key length at equivalent security levels
|
||
|
||
The following output is a speed test of different encryption algorithms
|
||
|
||
```bash
|
||
~ ❯ openssl speed rsa1024 rsa2048 rsa4096 ecdsap256 ed25519
|
||
Doing 1024 bits private rsa's for 10s: 73191 1024 bits private RSA's in 9.99s
|
||
Doing 1024 bits public rsa's for 10s: 1161066 1024 bits public RSA's in 9.99s
|
||
Doing 2048 bits private rsa's for 10s: 11346 2048 bits private RSA's in 9.99s
|
||
Doing 2048 bits public rsa's for 10s: 389696 2048 bits public RSA's in 9.99s
|
||
Doing 4096 bits private rsa's for 10s: 1643 4096 bits private RSA's in 10.00s
|
||
Doing 4096 bits public rsa's for 10s: 112285 4096 bits public RSA's in 9.99s
|
||
Doing 256 bits sign ecdsa's for 10s: 294504 256 bits ECDSA signs in 9.85s
|
||
Doing 256 bits verify ecdsa's for 10s: 101912 256 bits ECDSA verify in 9.99s
|
||
Doing 253 bits sign Ed25519's for 10s: 147102 253 bits Ed25519 signs in 9.99s
|
||
Doing 253 bits verify Ed25519's for 10s: 53317 253 bits Ed25519 verify in 10.00s
|
||
version: 3.0.13
|
||
...
|
||
|
||
...
|
||
sign verify sign/s verify/s
|
||
rsa 1024 bits 0.000136s 0.000009s 7326.4 116222.8
|
||
rsa 2048 bits 0.000880s 0.000026s 1135.7 39008.6
|
||
rsa 4096 bits 0.006086s 0.000089s 164.3 11239.7
|
||
sign verify sign/s verify/s
|
||
256 bits ecdsa (nistp256) 0.0000s 0.0001s 29898.9 10201.4
|
||
sign verify sign/s verify/s
|
||
253 bits EdDSA (Ed25519) 0.0001s 0.0002s 14724.9 5331.7
|
||
```
|
||
|
||
in my testing it seems as though ecdsa is actually a touch faster than ed25519 but my
|
||
testing is in no way scientific or thorough enough Ill leave that to the Pros.
|
||
|
||
`RSA 4096` is considered to be quite secure however the speed difference compared to
|
||
the more modern algorithms is considerably longer. Each algorithm has its strengths
|
||
and weaknesses, so research which works best for you.
|
||
|
||
In a homelab environment how noticeable this would actually be im not sure. We are
|
||
talking tiny fracions of a second.
|
||
|
||
for more insight: [https://serverfault.com/questions/1081266/ssh-keys-ed25519-vs-rsa-performance-demystified](https://serverfault.com/questions/1081266/ssh-keys-ed25519-vs-rsa-performance-demystified)
|
||
{{< /details >}}
|
||
|
||
When creating the SSH key you will be presented with a two questions.
|
||
|
||
- Location of the key
|
||
- Password
|
||
|
||
The default location of the key `$HOME/.ssh/id_ed25519` is fine for me.
|
||
|
||
I would strongly advise setting a good password for you SSH keys.
|
||
|
||
Now we can start copying these keys to remote hosts. The following command
|
||
will copy the public key to the remote host.
|
||
|
||
```
|
||
ssh-copy-id user@remote-host
|
||
```
|
||
|
||
{{< alert "bell" >}}
|
||
Change `user` and `remote-host` to your own values
|
||
{{< /alert >}}
|
||
|
||
If that succeeded then we should be anble to connect to the remote host using our ssh key
|
||
|
||
Hoping that you set a password for your ssh key, when attempting to connect the password prompt
|
||
will look slightly different. Instead of asking for the remote users password its asking for your
|
||
SSH keys password.
|
||
|
||
And just like that your system is a lot more secure.
|
||
|
||
## SSH key distribution
|
||
|
||
One pain point with SSH keys is when you have multiple ssh keys and multiple remote servers.
|
||
Having to distribute these keys can be time consuming, and honestly a waste of time.
|
||
|
||
One method thats easy to get up and running with minimal effort is using Github. We can add our
|
||
public SSH keys to github and run a command on our servers to pull those public keys.
|
||
|
||
{{< alert "bell" >}}
|
||
There are more advanced ways to get around this such as [SSH Certificate Authorities](posts/ssh-certificate-authority)
|
||
but thats out of the scope for this post.
|
||
{{< /alert >}}
|
||
|
||
To add an SSH key to github go to [github.com](https://github.com) and navigate to the SSH and GPG keys page
|
||
|
||
`Profile Icon > Settings > SSH and GPG keys > New SSH key`
|
||
|
||
[quicklink](https://github.com/settings/ssh/new)
|
||
|
||
- Title: Give a descriptive name for the key
|
||
- Key Type: Authentication key
|
||
- Key: Paste in the public key
|
||
|
||
to get your public key
|
||
|
||
```
|
||
cat $HOME/.ssh/id_ed25519.pub
|
||
```
|
||
|
||
Finally click `Add SSH Key`
|
||
|
||
### Importing the keys
|
||
|
||
When we are on the remote host we can run the following command to import our public ssh keys from github
|
||
Make sure the username is YOUR username.
|
||
|
||
```
|
||
ssh-import-id-gh username
|
||
```
|
||
|
||
## SSH Agent
|
||
|
||
Now we can setup the ssh agent to enable us to keep ssh-keys "logged-in" per session reducing the amount
|
||
we need to enter our password each time we want to spawn a new ssh session.
|
||
|
||
I make use of a bash script which is called in my `.zshrc` or `.bashrc`. So each time I spawn a new session
|
||
(Usually computer restart) it will spawn the ssh-agent and I can unlock my key for later use.
|
||
|
||
```bash
|
||
##### START Fix for ssh-agent #####
|
||
# Orig Ref: http://mah.everybody.org/docs/ssh
|
||
# Ref: https://web.archive.org/web/20220513101246/http://mah.everybody.org/docs/ssh
|
||
-
|
||
SSH_ENV="$HOME/.ssh/environment"
|
||
-
|
||
function start_agent {
|
||
echo "Initialising new SSH agent..."
|
||
/usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
|
||
echo succeeded
|
||
chmod 600 "${SSH_ENV}"
|
||
. "${SSH_ENV}" > /dev/null
|
||
/usr/bin/ssh-add;
|
||
}
|
||
-
|
||
# Source SSH settings, if applicable
|
||
if [ -f "${SSH_ENV}" ]; then
|
||
. "${SSH_ENV}" > /dev/null
|
||
#ps ${SSH_AGENT_PID} doesn't work under cywgin
|
||
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
|
||
start_agent;
|
||
}
|
||
else
|
||
start_agent;
|
||
fi
|
||
##### END Fix for ssh-agent #####
|
||
```
|
||
|
||
The next time your .rc file is sourced you will be asked for your ssh key password.
|
||
|
||
## Modify SSH Config
|
||
|
||
{{< alert "bell" >}}
|
||
Im using `vim`, if you are unfamilliar with vim you may find it easier to use `nano`
|
||
instead. If you want to try vim just remember you start in command mode, to insert
|
||
text you can press `i` to enter insert mode. When ready to save and quit press `esc`
|
||
to go back to command mode and type `:wq` the colon allows you to write commands,
|
||
`w` represents write and `q` represents quit.
|
||
{{< /alert >}}
|
||
|
||
```
|
||
sudo vim /etc/ssh/sshd_config
|
||
```
|
||
|
||
Some options are already in the file, some just need uncommenting and some you will
|
||
need to add yourself.
|
||
|
||
```
|
||
Protocol 2
|
||
Port 2222
|
||
PermitRootLogin no
|
||
MaxAuthTries 5
|
||
PasswordAuthentication no
|
||
PermitEmptyPasswords no
|
||
AllowUsers username username2
|
||
ClientAliveInterval 180
|
||
ClientAliveCountMax 1
|
||
```
|
||
|
||
- Protocol 2 - is the latest and most secure ssh protocol version
|
||
- Port 2222 - change the default port from 22 to 2222
|
||
- Stops port scanners on the internet hunting for default open ports
|
||
- PermitRootLogin no - Prevents the root user being able to login via ssh
|
||
- MaxAuthTries 5 - The number of allowed authentication attempts per connection, i.e if an attacker is trying to brute force a server after 3 failed attempts - or half of the set value - these failed attempts will be logged (useful to know for fail2ban) and then after 4 get locked out.
|
||
- PasswordAuthentication no - Disable password authentication since we use SSH keys now !!
|
||
- PermitEmptyPasswords no - may be redundant since we set no password authentication …
|
||
- AllowUsers - only allow certain users to use SSH
|
||
- ClientAliveInterval 180 - SSH daemon will send a packet to the client every 180s
|
||
- ClientAliveCountMax 1 - when the ClientAliveInterval packet is sent if the client does not respond the session is closed. If this value is 2 for example the SSH daemon will have a fault tolerance of 2 meaning that the client has to miss 2 packets each 180s apart before the session closes.
|
||
|
||
With the current ClientAliveInterval and ClientAliveCountMax you have 3 minutes of "downtime" before the
|
||
session is dropped. Downtime could be the internet connection dropped or you closed your laptop lid and
|
||
the machine goes to sleep. Essentially any state where the client machine cannot respond to the remote
|
||
servers "ping" every 3 minutes.
|
||
|
||
If 3 minutes is too little time for you, you may increase the interval or increase the ClientAliveCountMax
|
||
value.
|
||
|
||
## Fail2Ban
|
||
|
||
Fail2Ban is an IPS (Intrusion Protection System) which monitors ssh logs looking for failed login attempts
|
||
If multiple failed login attempts are detected then the source IP address is blocked in the firewall.
|
||
|
||
### Install Fail2Ban
|
||
|
||
```
|
||
sudo apt install fail2ban
|
||
```
|
||
|
||
### Fail2Ban Configuration
|
||
|
||
```
|
||
sudo vim /etc/fail2ban/jail.local
|
||
```
|
||
|
||
I use the following configuration
|
||
|
||
```fail2ban
|
||
[DEFAULT]
|
||
bantime = 8h
|
||
ignoreip = 127.0.0.1/8 xxx.xxx.xxx.xxx/xx
|
||
ignoreself = true
|
||
[sshd]
|
||
enabled = true
|
||
port = 2222
|
||
filter = sshd
|
||
logpath = /var/log/auth.log
|
||
maxretry = 2
|
||
```
|
||
|
||
- bantime - how long to block network connections for
|
||
- ignoreip - IP addresses / ranges that are ignored by fail2ban useful if you have static IPs set on your local machines or want to ignore your whole local network.
|
||
- ignoreself - dont ban itself xD
|
||
- sshd
|
||
- enabled - enables log monitoring for ssh
|
||
- port - the port we are using for ssh
|
||
- logpath - where your ssh logs are
|
||
- maxretry - how many retries before a connection is blocked
|
||
|
||
{{< alert "bell" >}}
|
||
maxretry in fail2ban is set to 2 because in our sshd_config file we set MaxAuthTries
|
||
to 5. So SSH will start to log after 3 failed attempts. When we hit 5 failed
|
||
attempt fail2Ban sees 2 failed attempts so it will block network connections.
|
||
So at 5 failed attempts **TOTAL** fail2ban blocks connections
|
||
{{< /alert >}}
|
||
|
||
these settings can be a bit aggressive in some situations so feel free to make them
|
||
a bit more relaxed however with our SSH keys setup properly we should not be
|
||
getting failed login attempts so this is _in my opinion_ already quite relaxed.
|
||
|
||
## Firewall
|
||
|
||
One thing I would like to mention here before we continue is that our network
|
||
security should not only be hardened on the edge. Each layer of our network
|
||
should be secured. The reason for this practice is that if an attacker does
|
||
manage to break through your edge then you can still isolate them and protect
|
||
resources from within the network. So even though we may have an edge firewall
|
||
we should consider setting up a firewall on each of our servers.
|
||
|
||
On Ubuntu server `ufw` is installed by default, thats what Ill be going over here.
|
||
You can install `ufw` on other distributions however they usually have default
|
||
firewall implementations like `iptables`
|
||
|
||
By Default the firewall on ubuntu servers is disabled. To check this:
|
||
|
||
```
|
||
sudo ufw status
|
||
```
|
||
|
||
{{< alert "bell" >}}
|
||
While its inactive we dont need to worry about locking ourselves out however
|
||
be aware that once your firewall is active any incoming traffic will be dropped
|
||
if we do not create accept rules for them.
|
||
{{< /alert >}}
|
||
|
||
We are going to create 2 catch all rules.
|
||
|
||
```
|
||
sudo ufw default allow outgoing
|
||
sudo ufw default deny incoming
|
||
```
|
||
|
||
- Allow all traffic going out of the server
|
||
- Deny all traffic coming into the server
|
||
|
||
Create a rule to allow our SSH traffic, make sure you set the port number you use to ssh
|
||
|
||
```
|
||
sudo ufw allow 2222
|
||
```
|
||
|
||
If you need any other incoming ports open, do that here.
|
||
|
||
We can now enable the firewall.
|
||
|
||
```
|
||
sudo ufw enable
|
||
```
|
||
|
||
check the status
|
||
|
||
```
|
||
sudo ufw status
|
||
```
|
||
|
||
## SSH Config
|
||
|
||
When we start using unconventional settings such as port 2222 it can be useful
|
||
to setup an ssh config file on our client machines.
|
||
|
||
My config files looks something like
|
||
This makes any host on my local network use the username and 2222.
|
||
I also have an alias for my VPS server.
|
||
|
||
```
|
||
Host 10.10.100.*
|
||
User username
|
||
Port 2222
|
||
|
||
Host vps-deb13-2c8g
|
||
HostName ipaddress-specific-server
|
||
User username
|
||
Port 2222
|
||
```
|
||
|
||
## 2 Factor Authentication - Optional
|
||
|
||
We can use 2 factor authentication on our servers much like you would with
|
||
something like github. By enabling the TOTP pam module when we login to the
|
||
server it will ask for a Time Based TOTP code.
|
||
|
||
These keys are time based and so it's vital that the time on the server and
|
||
client are correct.
|
||
|
||
The module I found is developed by Google and is open source
|
||
[google/google-authenticator-libpam](https://github.com/google/google-authenticator-libpam)
|
||
|
||
Install the module
|
||
|
||
```
|
||
sudo apt install libpam-google-authenticator
|
||
```
|
||
|
||
Edit the config
|
||
|
||
```
|
||
sudo vim /etc/pam.d/sshd
|
||
```
|
||
|
||
Comment out the following line
|
||
|
||
```
|
||
# @include common-auth
|
||
```
|
||
|
||
If we do not uncomment this then even though we are using SSH keys we will be
|
||
asked to use our user password to authenticate too.
|
||
|
||
Add the following line to the bottom of the file
|
||
|
||
```
|
||
auth required pam_google_authenticator.so
|
||
```
|
||
|
||
{{< alert "bell" >}}
|
||
If you have multiple users then it may be worth adding the nullok option as
|
||
this will prevent users who have not setup TOTP from being locked out.
|
||
|
||
```
|
||
auth required pam_google_authenticator.so nullok
|
||
```
|
||
|
||
{{< /alert >}}
|
||
|
||
Edit the ssh config
|
||
|
||
```
|
||
sudo vim /etc/ssh/sshd_config
|
||
```
|
||
|
||
Edit / Add the following
|
||
|
||
```
|
||
KbdInteractiveAuthentication yes
|
||
AuthenticationMethods publickey,keyboard-interactive
|
||
```
|
||
|
||
We are now ready to generte the TOTP codes for the user, run the following
|
||
|
||
```
|
||
google-authenticator
|
||
```
|
||
|
||
- `Do you want authentication tokens to be time-based (y/n)` - y
|
||
- Scan the Code into your Authenticator app or if you have a password manager use that :D
|
||
- You will be asked to enter a TOTP from the authenticator app
|
||
- You will be given recovery codes, save them somewhere
|
||
- `Do you want me to update your "/home/user/.google-authenticator" file ? (y/n)` - y
|
||
- The following questions are your choice however I answered
|
||
- `Do you want Disallow multiple uses of the same authentication token … (y/n)` - y
|
||
- `… In order to compensate for possible time-skew … (y/n)` - n (if you do experience problems you may change this option)
|
||
- `… 3 Login attempts every 30s … (y/n)` - y
|
||
|
||
|
||
|
||
## Restart
|
||
|
||
Now is the moment of truth, restarting the sshd service and fail2ban
|
||
|
||
```
|
||
sudo service ssh restart
|
||
sudo systemctl restart sshd.service
|
||
sudo systemctl restart fail2ban
|
||
```
|
||
|
||
Once you are comfortable you have everything setup I would suggest a full system
|
||
restart as updates were performed earlier too.
|
||
|
||
## Conclusion
|
||
|
||
This is a great start to improving security on your remote servers. But it's
|
||
only the start. There are still many things you could change / implement to
|
||
further increase security.
|
||
|
||
There are also network considerations like VLANs and ensuring solid firewall
|
||
rules.
|
||
|
||
As mentioned earlier in the post you could implement overlay networks such as
|
||
Netbird or Tailscale when working over the internet instead of exposing all
|
||
the ports.
|