blog-ahmza-com/content/posts/hardening-linux/index.md at 87867b9f304dcbd9056cda8bbbfc7e0a28d31ce4

ahmza/blog-ahmza-com

Fork 0

Files

T

ahmza 3322229b97 Updated hardening-linux post

2026-02-15 23:41:22 +00:00

18 KiB

Raw Blame History

title, description, tags, author, date, draft, weight, featureimage, lastmod

title

description

Introduction

This guide will go over the basic steps of hardening a Linux machine with a focus on remote access. The reason this is important is because when we begin to expose services to the internet and make them publicly accessible then that means anyone can probe around.

If you have not heard of Overlay Networks before they're similar to VPNs however do not need any open ports as they use NAT Punching.

Check out my post on Netbird an alternative to Tailscale

Follow along and get your remote systems secure !!

Initial Steps

Before we get into the meat and potatoes of this guide, it would be a good idea to go over the following steps, especially if you are configuring a new install.

Software Updates

Software updates are key for security as they will often include security patches. Depending on your flavour of Linux this command will vary.

sudo apt update && sudo apt upgrade

{{< alert "bell" >}} If you do not expect to be logging into this remote machine very often then using Ansible to update Remote Machines is a great alternative to the manual approach {{< /alert >}}

User Accounts

It's not a good idea to continue using the root account after install, reason being the root account has complete control over the system, whereas a user account is more restricted by default. Logging is more granular with different user accounts, especially with more than one user.

The following command will allow you to create a new user, it will also ask you some questions about user information however the only required fields are the username and password.

adduser username

now we can add our newly created user to the sudo group to give admin permissions

usermod -aG sudo username

{{< alert "bell" >}} Remember to replace username with your desired username {{< /alert >}}

you can now logout of root and log back in as your user. Further action on the server should now be executed from this user account.

Creating SSH Keys

Now that you have your own user account, we can create an SSH key for it. An SSH Key is a Public and Private key pair which allow client and server to authenticate using Public Key cryptography. Public Key cryptography is much harder to crack compared to a password, also we no longer send passwords across networks which can be at risk of interception.

SSH can also be more convenient once you set up ssh-agent which will be covered later in the guide, with password authentication you need to enter a password everytime, however with ssh-agent you can enter your ssh-key password once and it remains unlocked for the remainder of the session.

{{< alert "fire" >}} IMPORTANT: NEVER share your private ssh key ... it's private for a reason. {{< /alert >}}

By running the following command you will create an ssh-key !!

ssh-keygen -t ed25519 -C NameYourKey

-t - specifies the type of cryptographic algorithm we will use
-C - notice it's capital, A name for your ssh keys. I usually give them the hostname of the local machine

{{< alert "bell" >}} If you would like a deep dive into why I pick this algorithm, click the drop down, else skip ahead :D {{< /alert >}} {{< details >}} The -t flag specifies the type of encryption to use. The reasons I have gone with ed25519:

ed25519 is faster and generation, signing and verification operations
Believed to be more resistant to certain attacks
shorter key length at equivalent security levels

The following output is a speed test of different encryption algorithms

~ ❯ openssl speed rsa1024 rsa2048 rsa4096 ecdsap256 ed25519
Doing 1024 bits private rsa's for 10s: 73191 1024 bits private RSA's in 9.99s
Doing 1024 bits public rsa's for 10s: 1161066 1024 bits public RSA's in 9.99s
Doing 2048 bits private rsa's for 10s: 11346 2048 bits private RSA's in 9.99s
Doing 2048 bits public rsa's for 10s: 389696 2048 bits public RSA's in 9.99s
Doing 4096 bits private rsa's for 10s: 1643 4096 bits private RSA's in 10.00s
Doing 4096 bits public rsa's for 10s: 112285 4096 bits public RSA's in 9.99s
Doing 256 bits sign ecdsa's for 10s: 294504 256 bits ECDSA signs in 9.85s
Doing 256 bits verify ecdsa's for 10s: 101912 256 bits ECDSA verify in 9.99s
Doing 253 bits sign Ed25519's for 10s: 147102 253 bits Ed25519 signs in 9.99s
Doing 253 bits verify Ed25519's for 10s: 53317 253 bits Ed25519 verify in 10.00s
version: 3.0.13
...

...
                  sign    verify    sign/s verify/s
rsa 1024 bits 0.000136s 0.000009s   7326.4 116222.8
rsa 2048 bits 0.000880s 0.000026s   1135.7  39008.6
rsa 4096 bits 0.006086s 0.000089s    164.3  11239.7
                              sign    verify    sign/s verify/s
 256 bits ecdsa (nistp256)   0.0000s   0.0001s  29898.9  10201.4
                              sign    verify    sign/s verify/s
 253 bits EdDSA (Ed25519)   0.0001s   0.0002s  14724.9   5331.7

in my testing it seems as though ecdsa is actually a touch faster than ed25519 but my testing is in no way scientific or thorough enough Ill leave that to the Pros.

RSA 4096 is considered to be quite secure however the speed difference compared to the more modern algorithms is considerably longer. Each algorithm has its strengths and weaknesses, so research which works best for you.

In a homelab environment how noticeable this would actually be im not sure. We are talking tiny fracions of a second.

for more insight: https://serverfault.com/questions/1081266/ssh-keys-ed25519-vs-rsa-performance-demystified {{< /details >}}

When creating the SSH key you will be presented with a two questions.

Location of the key
Password

The default location of the key $HOME/.ssh/id_ed25519 is fine for me.

I would strongly advise setting a good password for you SSH keys.

Now we can start copying these keys to remote hosts. The following command will copy the public key to the remote host.

ssh-copy-id user@remote-host

{{< alert "bell" >}} Change user and remote-host to your own values {{< /alert >}}

If that succeeded then we should be anble to connect to the remote host using our ssh key

Hoping that you set a password for your ssh key, when attempting to connect the password prompt will look slightly different. Instead of asking for the remote users password its asking for your SSH keys password.

And just like that your system is a lot more secure.

SSH key distribution

One pain point with SSH keys is when you have multiple ssh keys and multiple remote servers. Having to distribute these keys can be time consuming, and honestly a waste of time.

One method thats easy to get up and running with minimal effort is using Github. We can add our public SSH keys to github and run a command on our servers to pull those public keys.

{{< alert "bell" >}} There are more advanced ways to get around this such as SSH Certificate Authorities but thats out of the scope for this post. {{< /alert >}}

To add an SSH key to github go to github.com and navigate to the SSH and GPG keys page

Profile Icon > Settings > SSH and GPG keys > New SSH key

quicklink

Title: Give a descriptive name for the key
Key Type: Authentication key
Key: Paste in the public key

to get your public key

cat $HOME/.ssh/id_ed25519.pub

Finally click Add SSH Key

Importing the keys

When we are on the remote host we can run the following command to import our public ssh keys from github Make sure the username is YOUR username.

ssh-import-id-gh username

SSH Agent

Now we can setup the ssh agent to enable us to keep ssh-keys "logged-in" per session reducing the amount we need to enter our password each time we want to spawn a new ssh session.

I make use of a bash script which is called in my .zshrc or .bashrc. So each time I spawn a new session (Usually computer restart) it will spawn the ssh-agent and I can unlock my key for later use.

##### START Fix for ssh-agent #####
# Orig Ref: http://mah.everybody.org/docs/ssh
# Ref: https://web.archive.org/web/20220513101246/http://mah.everybody.org/docs/ssh
-
SSH_ENV="$HOME/.ssh/environment"
-
function start_agent {
    echo "Initialising new SSH agent..."
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add;
}
-
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    #ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi
##### END Fix for ssh-agent #####

The next time your .rc file is sourced you will be asked for your ssh key password.

Modify SSH Config

{{< alert "bell" >}} Im using vim, if you are unfamilliar with vim you may find it easier to use nano instead. If you want to try vim just remember you start in command mode, to insert text you can press i to enter insert mode. When ready to save and quit press esc to go back to command mode and type :wq the colon allows you to write commands, w represents write and q represents quit. {{< /alert >}}

sudo vim /etc/ssh/sshd_config

Some options are already in the file, some just need uncommenting and some you will need to add yourself.

Protocol 2
Port 2222
PermitRootLogin no
MaxAuthTries 5
PasswordAuthentication no
PermitEmptyPasswords no
AllowUsers username username2
ClientAliveInterval 180
ClientAliveCountMax 1

Protocol 2 - is the latest and most secure ssh protocol version
Port 2222 - change the default port from 22 to 2222
- Stops port scanners on the internet hunting for default open ports
PermitRootLogin no - Prevents the root user being able to login via ssh
MaxAuthTries 5 - The number of allowed authentication attempts per connection, i.e if an attacker is trying to brute force a server after 3 failed attempts - or half of the set value - these failed attempts will be logged (useful to know for fail2ban) and then after 4 get locked out.
PasswordAuthentication no - Disable password authentication since we use SSH keys now !!
PermitEmptyPasswords no - may be redundant since we set no password authentication …
AllowUsers - only allow certain users to use SSH
ClientAliveInterval 180 - SSH daemon will send a packet to the client every 180s
ClientAliveCountMax 1 - when the ClientAliveInterval packet is sent if the client does not respond the session is closed. If this value is 2 for example the SSH daemon will have a fault tolerance of 2 meaning that the client has to miss 2 packets each 180s apart before the session closes.

With the current ClientAliveInterval and ClientAliveCountMax you have 3 minutes of "downtime" before the session is dropped. Downtime could be the internet connection dropped or you closed your laptop lid and the machine goes to sleep. Essentially any state where the client machine cannot respond to the remote servers "ping" every 3 minutes.

If 3 minutes is too little time for you, you may increase the interval or increase the ClientAliveCountMax value.

Fail2Ban

Fail2Ban is an IPS (Intrusion Protection System) which monitors ssh logs looking for failed login attempts If multiple failed login attempts are detected then the source IP address is blocked in the firewall.

Install Fail2Ban

sudo apt install fail2ban

Fail2Ban Configuration

sudo vim /etc/fail2ban/jail.local

I use the following configuration

[DEFAULT]
  bantime = 8h
  ignoreip = 127.0.0.1/8 xxx.xxx.xxx.xxx/xx
  ignoreself = true
  [sshd]
  enabled = true
  port = 2222
  filter = sshd
  logpath = /var/log/auth.log
  maxretry = 2

bantime - how long to block network connections for
ignoreip - IP addresses / ranges that are ignored by fail2ban useful if you have static IPs set on your local machines or want to ignore your whole local network.
ignoreself - dont ban itself xD
sshd
- enabled - enables log monitoring for ssh
- port - the port we are using for ssh
- logpath - where your ssh logs are
- maxretry - how many retries before a connection is blocked

{{< alert "bell" >}} maxretry in fail2ban is set to 2 because in our sshd_config file we set MaxAuthTries to 5. So SSH will start to log after 3 failed attempts. When we hit 5 failed attempt fail2Ban sees 2 failed attempts so it will block network connections. So at 5 failed attempts TOTAL fail2ban blocks connections {{< /alert >}}

these settings can be a bit aggressive in some situations so feel free to make them a bit more relaxed however with our SSH keys setup properly we should not be getting failed login attempts so this is in my opinion already quite relaxed.

Firewall

One thing I would like to mention here before we continue is that our network security should not only be hardened on the edge. Each layer of our network should be secured. The reason for this practice is that if an attacker does manage to break through your edge then you can still isolate them and protect resources from within the network. So even though we may have an edge firewall we should consider setting up a firewall on each of our servers.

On Ubuntu server ufw is installed by default, thats what Ill be going over here. You can install ufw on other distributions however they usually have default firewall implementations like iptables

By Default the firewall on ubuntu servers is disabled. To check this:

sudo ufw status

{{< alert "bell" >}} While its inactive we dont need to worry about locking ourselves out however be aware that once your firewall is active any incoming traffic will be dropped if we do not create accept rules for them. {{< /alert >}}

We are going to create 2 catch all rules.

sudo ufw default allow outgoing
sudo ufw default deny incoming

Allow all traffic going out of the server
Deny all traffic coming into the server

Create a rule to allow our SSH traffic, make sure you set the port number you use to ssh

sudo ufw allow 2222

If you need any other incoming ports open, do that here.

We can now enable the firewall.

sudo ufw enable

check the status

sudo ufw status

SSH Config

When we start using unconventional settings such as port 2222 it can be useful to setup an ssh config file on our client machines.

My config files looks something like This makes any host on my local network use the username and 2222. I also have an alias for my VPS server.

Host 10.10.100.*
    User username
    Port 2222

Host vps-deb13-2c8g
    HostName ipaddress-specific-server
    User username
    Port 2222

2 Factor Authentication - Optional

We can use 2 factor authentication on our servers much like you would with something like github. By enabling the TOTP pam module when we login to the server it will ask for a Time Based TOTP code.

These keys are time based and so it's vital that the time on the server and client are correct.

The module I found is developed by Google and is open source google/google-authenticator-libpam

Install the module

sudo apt install libpam-google-authenticator

Edit the config

sudo vim /etc/pam.d/sshd

Comment out the following line

# @include common-auth

If we do not uncomment this then even though we are using SSH keys we will be asked to use our user password to authenticate too.

Add the following line to the bottom of the file

auth required pam_google_authenticator.so

{{< alert "bell" >}} If you have multiple users then it may be worth adding the nullok option as this will prevent users who have not setup TOTP from being locked out.

auth required pam_google_authenticator.so nullok

Edit the ssh config

sudo vim /etc/ssh/sshd_config

Edit / Add the following

KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

We are now ready to generte the TOTP codes for the user, run the following

google-authenticator

Do you want authentication tokens to be time-based (y/n) - y
Scan the Code into your Authenticator app or if you have a password manager use that :D
You will be asked to enter a TOTP from the authenticator app
You will be given recovery codes, save them somewhere
Do you want me to update your "/home/user/.google-authenticator" file ? (y/n) - y
The following questions are your choice however I answered
Do you want Disallow multiple uses of the same authentication token … (y/n) - y
… In order to compensate for possible time-skew … (y/n) - n (if you do experience problems you may change this option)
… 3 Login attempts every 30s … (y/n) - y

Restart

Now is the moment of truth, restarting the sshd service and fail2ban

sudo service ssh restart
sudo systemctl restart sshd.service
sudo systemctl restart fail2ban

Once you are comfortable you have everything setup I would suggest a full system restart as updates were performed earlier too.

Conclusion

This is a great start to improving security on your remote servers. But it's only the start. There are still many things you could change / implement to further increase security.

There are also network considerations like VLANs and ensuring solid firewall rules.

As mentioned earlier in the post you could implement overlay networks such as Netbird or Tailscale when working over the internet instead of exposing all the ports.

18 KiB Raw Blame History Unescape Escape