diff --git a/content/posts/hardening-linux/google-authenticator-cli.webp b/content/posts/hardening-linux/google-authenticator-cli.webp new file mode 100644 index 0000000..2738486 Binary files /dev/null and b/content/posts/hardening-linux/google-authenticator-cli.webp differ diff --git a/content/posts/hardening-linux/hardening-linux.md b/content/posts/hardening-linux/hardening-linux.md deleted file mode 100644 index 5a05e22..0000000 --- a/content/posts/hardening-linux/hardening-linux.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Hardening Linux - An Update to Securing SSH -description: "A Guide to Making Linux a Little more secure" -tags: ["Linux", "Ubuntu"] -author: "ahmza" -date: 2025-06-05T15:34:10+01:00 -draft: true -weight: -featureimage: -lastmod: ---- - ---- -{{< alert icon="fire" >}} -> - This Guide assumes a Linux -> Linux SSH Session -> - Please take care with production systems -> - I take no responsibility for misconfigured systems and lockouts -> - Try in a VM first and thank me later :) -> - My Commands will be for Ubuntu Server 24.04 -{{< /alert >}} ---- - -## Introduction -This guide will go over the basic steps of hardening a Linux machine with a -focus on remote access. The reason this is important is because when we begin to -expose services to the internet and make them publicly accessible then that means -anyone can probe around. - -There are better ways around this especially if you are the only user, or If -your users dont mind taking an extra step before being able to access your -resources. If you have not heard of _Overlay Networks_ before they're similar to -VPNs however do not need any open ports as they use NAT Punching AKA Hole -Punching. - -Check out my post on [Tailscale](/posts/no-more-port-forwarding/) - -regardless, it's always a good idea to implement good security practice so -follow along and get your remote systems secure !! - -## Initial Steps - -Before we get into the meat and potatos of this guide, it would be a good idea -to go over the following steps, especially if you are configuring a new install. - -### Software Updates -Software updates are key for security as they will often include security -patches. Depending on your flavour of Linux this command will vary. - -```bash -sudo apt update && sudo apt upgrade -``` -{{< alert "bell" >}} -If you do not expect to be logging into this remote machine very often it may be -a good idea to [setup automatic updates](/posts/setting-up-automatic-updates) -however if you want more control then using [Ansible to update Remote Machines](/posts/ansible-to-manage-remote-servers) -would be a better option. -{{< /alert >}} - -### User Accounts -It's not a good idea to continue to use a root account after install, it's much -more secure to setup another user account with sudo access. The reason is the -root account has complete control over the system, where a user account is more -restricted by default. As well as this logging is more granular with different -user accounts. - -The following command will allow you to create a new user, it will also ask you -some questions about user information however the only required fields are the -username and password if you want one. - -```bash -adduser username -``` - -now we can add our newly created user to the sudo group to give admin permissions - -``` bash -usermod -aG sudo username -``` - -{{< alert "bell" >}} -Remember to replace username with your desired username -{{< /alert >}} - -you can now logout of root and log back in as your user. Further action on the -server should now be executed from this user account. - -## Creating SSH Keys -Now that you have your own user account, we can create an SSH key for it. An SSH -Key is a Public and Private key pair which allow client and server to -authenticate using Public Key cryptography. Public Key cryptography is much -harder to crack compared to a password, as well as this we no longer need to -send passwords across networks which can be at risk of interception. - -SSH can also be more convenient once you set up ssh-agent which will be covered -later in the guide, with password authentication you need to enter a password -everytime, however with ssh-agent you can enter your ssh-key password once and -it remains unlocked for the remainder of the session. - -{{< alert "fire" >}} -**IMPORTANT:** **NEVER** share your private ssh key ... it's private for a reason. -{{< /alert >}} - -By running the following command you will create an ssh-key !! -```bash -ssh-keygen -t ed25519 -C NameYourKey -``` -- `-t` - specifies the type of cryptographic algorithm we will use -- `-C` - notice it's capital, A name for your ssh keys. I usually give them the -hostname of the local machine - - - -## Conclusion diff --git a/content/posts/hardening-linux/index.md b/content/posts/hardening-linux/index.md new file mode 100644 index 0000000..fcccec9 --- /dev/null +++ b/content/posts/hardening-linux/index.md @@ -0,0 +1,551 @@ +--- +title: Hardening Linux - An Update to Securing SSH +description: "A Guide to Making Linux a Little more secure" +tags: ["Linux", "Ubuntu"] +author: "ahmza" +date: 2025-06-05T15:34:10+01:00 +draft: true +weight: +featureimage: +lastmod: +--- + +Update to [Securing SSH](/posts/securing-ssh/) + +{{< alert icon="fire" >}} + +> - This Guide assumes a Linux -> Linux SSH Session +> - Please take care with production systems +> - I take no responsibility for misconfigured systems and lockouts +> - Try in a VM first and thank me later :) +> - My Commands will be for Debian 13 Trixie + +{{< /alert >}} + +## Introduction + +This guide will go over the basic steps of hardening a Linux machine with a +focus on remote access. The reason this is important is because when we begin to +expose services to the internet and make them publicly accessible then that means +anyone can probe around. + +If you have not heard of _Overlay Networks_ before they're similar to +VPNs however do not need any open ports as they use NAT Punching. + +Check out my post on [Netbird](/posts/no-more-port-forwarding/#netbird) an alternative to [Tailscale](/posts/no-more-port-forwarding/#tailscale) + +Follow along and get your remote systems secure !! + +## Initial Steps + +Before we get into the meat and potatoes of this guide, it would be a good idea +to go over the following steps, especially if you are configuring a new install. + +### Software Updates + +Software updates are key for security as they will often include security +patches. Depending on your flavour of Linux this command will vary. + +```bash +sudo apt update && sudo apt upgrade +``` + +{{< alert "bell" >}} +If you do not expect to be logging into this remote machine very often +then using [Ansible to update Remote Machines](/posts/ansible-to-manage-remote-servers) +is a great alternative to the manual approach +{{< /alert >}} + +### User Accounts + +It's not a good idea to continue using the root account after install, reason being +the root account has complete control over the system, whereas a user account is more +restricted by default. Logging is more granular with different user accounts, +especially with more than one user. + +The following command will allow you to create a new user, it will also ask you +some questions about user information however the only required fields are the +username and password. + +```bash +adduser username +``` + +now we can add our newly created user to the sudo group to give admin permissions + +```bash +usermod -aG sudo username +``` + +{{< alert "bell" >}} +Remember to replace username with your desired username +{{< /alert >}} + +you can now logout of root and log back in as your user. Further action on the +server should now be executed from this user account. + +## Creating SSH Keys + +Now that you have your own user account, we can create an SSH key for it. An SSH +Key is a Public and Private key pair which allow client and server to +authenticate using Public Key cryptography. Public Key cryptography is much +harder to crack compared to a password, also we no longer send passwords across +networks which can be at risk of interception. + +SSH can also be more convenient once you set up ssh-agent which will be covered +later in the guide, with password authentication you need to enter a password +everytime, however with ssh-agent you can enter your ssh-key password once and +it remains unlocked for the remainder of the session. + +{{< alert "fire" >}} +**IMPORTANT:** **NEVER** share your private ssh key ... it's private for a reason. +{{< /alert >}} + +By running the following command you will create an ssh-key !! + +```bash +ssh-keygen -t ed25519 -C NameYourKey +``` + +- `-t` - specifies the type of cryptographic algorithm we will use +- `-C` - notice it's capital, A name for your ssh keys. I usually give them the + hostname of the local machine + +{{< alert "bell" >}} +If you would like a deep dive into why I pick this algorithm, click the drop down, else skip ahead :D +{{< /alert >}} +{{< details >}} +The `-t` flag specifies the type of encryption to use. The reasons I have gone with `ed25519`: + +- `ed25519` is faster and generation, signing and verification operations +- Believed to be more resistant to certain attacks +- shorter key length at equivalent security levels + +The following output is a speed test of different encryption algorithms + +```bash +~ ❯ openssl speed rsa1024 rsa2048 rsa4096 ecdsap256 ed25519 +Doing 1024 bits private rsa's for 10s: 73191 1024 bits private RSA's in 9.99s +Doing 1024 bits public rsa's for 10s: 1161066 1024 bits public RSA's in 9.99s +Doing 2048 bits private rsa's for 10s: 11346 2048 bits private RSA's in 9.99s +Doing 2048 bits public rsa's for 10s: 389696 2048 bits public RSA's in 9.99s +Doing 4096 bits private rsa's for 10s: 1643 4096 bits private RSA's in 10.00s +Doing 4096 bits public rsa's for 10s: 112285 4096 bits public RSA's in 9.99s +Doing 256 bits sign ecdsa's for 10s: 294504 256 bits ECDSA signs in 9.85s +Doing 256 bits verify ecdsa's for 10s: 101912 256 bits ECDSA verify in 9.99s +Doing 253 bits sign Ed25519's for 10s: 147102 253 bits Ed25519 signs in 9.99s +Doing 253 bits verify Ed25519's for 10s: 53317 253 bits Ed25519 verify in 10.00s +version: 3.0.13 +... + +... + sign verify sign/s verify/s +rsa 1024 bits 0.000136s 0.000009s 7326.4 116222.8 +rsa 2048 bits 0.000880s 0.000026s 1135.7 39008.6 +rsa 4096 bits 0.006086s 0.000089s 164.3 11239.7 + sign verify sign/s verify/s + 256 bits ecdsa (nistp256) 0.0000s 0.0001s 29898.9 10201.4 + sign verify sign/s verify/s + 253 bits EdDSA (Ed25519) 0.0001s 0.0002s 14724.9 5331.7 +``` + +in my testing it seems as though ecdsa is actually a touch faster than ed25519 but my +testing is in no way scientific or thorough enough Ill leave that to the Pros. + +`RSA 4096` is considered to be quite secure however the speed difference compared to +the more modern algorithms is considerably longer. Each algorithm has its strengths +and weaknesses, so research which works best for you. + +In a homelab environment how noticeable this would actually be im not sure. We are +talking tiny fracions of a second. + +for more insight: [https://serverfault.com/questions/1081266/ssh-keys-ed25519-vs-rsa-performance-demystified](https://serverfault.com/questions/1081266/ssh-keys-ed25519-vs-rsa-performance-demystified) +{{< /details >}} + +When creating the SSH key you will be presented with a two questions. + +- Location of the key +- Password + +The default location of the key `$HOME/.ssh/id_ed25519` is fine for me. + +I would strongly advise setting a good password for you SSH keys. + +Now we can start copying these keys to remote hosts. The following command +will copy the public key to the remote host. + +``` +ssh-copy-id user@remote-host +``` + +{{< alert "bell" >}} +Change `user` and `remote-host` to your own values +{{< /alert >}} + +If that succeeded then we should be anble to connect to the remote host using our ssh key + +Hoping that you set a password for your ssh key, when attempting to connect the password prompt +will look slightly different. Instead of asking for the remote users password its asking for your +SSH keys password. + +And just like that your system is a lot more secure. + +## SSH key distribution + +One pain point with SSH keys is when you have multiple ssh keys and multiple remote servers. +Having to distribute these keys can be time consuming, and honestly a waste of time. + +One method thats easy to get up and running with minimal effort is using Github. We can add our +public SSH keys to github and run a command on our servers to pull those public keys. + +{{< alert "bell" >}} +There are more advanced ways to get around this such as [SSH Certificate Authorities](posts/ssh-certificate-authority) +but thats out of the scope for this post. +{{< /alert >}} + +To add an SSH key to github go to [github.com](https://github.com) and navigate to the SSH and GPG keys page + +`Profile Icon > Settings > SSH and GPG keys > New SSH key` + +[quicklink](https://github.com/settings/ssh/new) + +- Title: Give a descriptive name for the key +- Key Type: Authentication key +- Key: Paste in the public key + +to get your public key + +``` +cat $HOME/.ssh/id_ed25519.pub +``` + +Finally click `Add SSH Key` + +### Importing the keys + +When we are on the remote host we can run the following command to import our public ssh keys from github +Make sure the username is YOUR username. + +``` +ssh-import-id-gh username +``` + +## SSH Agent + +Now we can setup the ssh agent to enable us to keep ssh-keys "logged-in" per session reducing the amount +we need to enter our password each time we want to spawn a new ssh session. + +I make use of a bash script which is called in my `.zshrc` or `.bashrc`. So each time I spawn a new session +(Usually computer restart) it will spawn the ssh-agent and I can unlock my key for later use. + +```bash +##### START Fix for ssh-agent ##### +# Orig Ref: http://mah.everybody.org/docs/ssh +# Ref: https://web.archive.org/web/20220513101246/http://mah.everybody.org/docs/ssh +- +SSH_ENV="$HOME/.ssh/environment" +- +function start_agent { + echo "Initialising new SSH agent..." + /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}" + echo succeeded + chmod 600 "${SSH_ENV}" + . "${SSH_ENV}" > /dev/null + /usr/bin/ssh-add; +} +- +# Source SSH settings, if applicable +if [ -f "${SSH_ENV}" ]; then + . "${SSH_ENV}" > /dev/null + #ps ${SSH_AGENT_PID} doesn't work under cywgin + ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || { + start_agent; + } +else + start_agent; +fi +##### END Fix for ssh-agent ##### +``` + +The next time your .rc file is sourced you will be asked for your ssh key password. + +## Modify SSH Config + +{{< alert "bell" >}} +Im using `vim`, if you are unfamilliar with vim you may find it easier to use `nano` +instead. If you want to try vim just remember you start in command mode, to insert +text you can press `i` to enter insert mode. When ready to save and quit press `esc` +to go back to command mode and type `:wq` the colon allows you to write commands, +`w` represents write and `q` represents quit. +{{< /alert >}} + +``` +sudo vim /etc/ssh/sshd_config +``` + +Some options are already in the file, some just need uncommenting and some you will +need to add yourself. + +``` +Protocol 2 +Port 2222 +PermitRootLogin no +MaxAuthTries 5 +PasswordAuthentication no +PermitEmptyPasswords no +AllowUsers username username2 +ClientAliveInterval 180 +ClientAliveCountMax 1 +``` + +- Protocol 2 - is the latest and most secure ssh protocol version +- Port 2222 - change the default port from 22 to 2222 + - Stops port scanners on the internet hunting for default open ports +- PermitRootLogin no - Prevents the root user being able to login via ssh +- MaxAuthTries 5 - The number of allowed authentication attempts per connection, i.e if an attacker is trying to brute force a server after 3 failed attempts - or half of the set value - these failed attempts will be logged (useful to know for fail2ban) and then after 4 get locked out. +- PasswordAuthentication no - Disable password authentication since we use SSH keys now !! +- PermitEmptyPasswords no - may be redundant since we set no password authentication … +- AllowUsers - only allow certain users to use SSH +- ClientAliveInterval 180 - SSH daemon will send a packet to the client every 180s +- ClientAliveCountMax 1 - when the ClientAliveInterval packet is sent if the client does not respond the session is closed. If this value is 2 for example the SSH daemon will have a fault tolerance of 2 meaning that the client has to miss 2 packets each 180s apart before the session closes. + +With the current ClientAliveInterval and ClientAliveCountMax you have 3 minutes of "downtime" before the +session is dropped. Downtime could be the internet connection dropped or you closed your laptop lid and +the machine goes to sleep. Essentially any state where the client machine cannot respond to the remote +servers "ping" every 3 minutes. + +If 3 minutes is too little time for you, you may increase the interval or increase the ClientAliveCountMax +value. + +## Fail2Ban + +Fail2Ban is an IPS (Intrusion Protection System) which monitors ssh logs looking for failed login attempts +If multiple failed login attempts are detected then the source IP address is blocked in the firewall. + +### Install Fail2Ban + +``` +sudo apt install fail2ban +``` + +### Fail2Ban Configuration + +``` +sudo vim /etc/fail2ban/jail.local +``` + +I use the following configuration + +```fail2ban +[DEFAULT] + bantime = 8h + ignoreip = 127.0.0.1/8 xxx.xxx.xxx.xxx/xx + ignoreself = true + [sshd] + enabled = true + port = 2222 + filter = sshd + logpath = /var/log/auth.log + maxretry = 2 +``` + +- bantime - how long to block network connections for +- ignoreip - IP addresses / ranges that are ignored by fail2ban useful if you have static IPs set on your local machines or want to ignore your whole local network. +- ignoreself - dont ban itself xD +- sshd + - enabled - enables log monitoring for ssh + - port - the port we are using for ssh + - logpath - where your ssh logs are + - maxretry - how many retries before a connection is blocked + +{{< alert "bell" >}} +maxretry in fail2ban is set to 2 because in our sshd_config file we set MaxAuthTries +to 5. So SSH will start to log after 3 failed attempts. When we hit 5 failed +attempt fail2Ban sees 2 failed attempts so it will block network connections. +So at 5 failed attempts **TOTAL** fail2ban blocks connections +{{< /alert >}} + +these settings can be a bit aggressive in some situations so feel free to make them +a bit more relaxed however with our SSH keys setup properly we should not be +getting failed login attempts so this is _in my opinion_ already quite relaxed. + +## Firewall + +One thing I would like to mention here before we continue is that our network +security should not only be hardened on the edge. Each layer of our network +should be secured. The reason for this practice is that if an attacker does +manage to break through your edge then you can still isolate them and protect +resources from within the network. So even though we may have an edge firewall +we should consider setting up a firewall on each of our servers. + +On Ubuntu server `ufw` is installed by default, thats what Ill be going over here. +You can install `ufw` on other distributions however they usually have default +firewall implementations like `iptables` + +By Default the firewall on ubuntu servers is disabled. To check this: + +``` +sudo ufw status +``` + +{{< alert "bell" >}} +While its inactive we dont need to worry about locking ourselves out however +be aware that once your firewall is active any incoming traffic will be dropped +if we do not create accept rules for them. +{{< /alert >}} + +We are going to create 2 catch all rules. + +``` +sudo ufw default allow outgoing +sudo ufw default deny incoming +``` + +- Allow all traffic going out of the server +- Deny all traffic coming into the server + +Create a rule to allow our SSH traffic, make sure you set the port number you use to ssh + +``` +sudo ufw allow 2222 +``` + +If you need any other incoming ports open, do that here. + +We can now enable the firewall. + +``` +sudo ufw enable +``` + +check the status + +``` +sudo ufw status +``` + +## SSH Config + +When we start using unconventional settings such as port 2222 it can be useful +to setup an ssh config file on our client machines. + +My config files looks something like +This makes any host on my local network use the username and 2222. +I also have an alias for my VPS server. + +``` +Host 10.10.100.* + User username + Port 2222 + +Host vps-deb13-2c8g + HostName ipaddress-specific-server + User username + Port 2222 +``` + +## 2 Factor Authentication - Optional + +We can use 2 factor authentication on our servers much like you would with +something like github. By enabling the TOTP pam module when we login to the +server it will ask for a Time Based TOTP code. + +These keys are time based and so it's vital that the time on the server and +client are correct. + +The module I found is developed by Google and is open source +[google/google-authenticator-libpam](https://github.com/google/google-authenticator-libpam) + +Install the module + +``` +sudo apt install libpam-google-authenticator +``` + +Edit the config + +``` +sudo vim /etc/pam.d/sshd +``` + +Comment out the following line + +``` +# @include common-auth +``` + +If we do not uncomment this then even though we are using SSH keys we will be +asked to use our user password to authenticate too. + +Add the following line to the bottom of the file + +``` +auth required pam_google_authenticator.so +``` + +{{< alert "bell" >}} +If you have multiple users then it may be worth adding the nullok option as +this will prevent users who have not setup TOTP from being locked out. + +``` +auth required pam_google_authenticator.so nullok +``` + +{{< /alert >}} + +Edit the ssh config + +``` +sudo vim /etc/ssh/sshd_config +``` + +Edit / Add the following + +``` +KbdInteractiveAuthentication yes +AuthenticationMethods publickey,keyboard-interactive +``` + +We are now ready to generte the TOTP codes for the user, run the following + +``` +google-authenticator +``` + +- `Do you want authentication tokens to be time-based (y/n)` - y +- Scan the Code into your Authenticator app or if you have a password manager use that :D +- You will be asked to enter a TOTP from the authenticator app +- You will be given recovery codes, save them somewhere +- `Do you want me to update your "/home/user/.google-authenticator" file ? (y/n)` - y +- The following questions are your choice however I answered +- `Do you want Disallow multiple uses of the same authentication token … (y/n)` - y +- `… In order to compensate for possible time-skew … (y/n)` - n (if you do experience problems you may change this option) +- `… 3 Login attempts every 30s … (y/n)` - y + +![google-authenticator-cli](google-authenticator-cli.webp) + +## Restart + +Now is the moment of truth, restarting the sshd service and fail2ban + +``` +sudo service ssh restart +sudo systemctl restart sshd.service +sudo systemctl restart fail2ban +``` + +Once you are comfortable you have everything setup I would suggest a full system +restart as updates were performed earlier too. + +## Conclusion + +This is a great start to improving security on your remote servers. But it's +only the start. There are still many things you could change / implement to +further increase security. + +There are also network considerations like VLANs and ensuring solid firewall +rules. + +As mentioned earlier in the post you could implement overlay networks such as +Netbird or Tailscale when working over the internet instead of exposing all +the ports.