ahmza/blog-ahmza-com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated content in old posts.

Browse Source
This commit is contained in:
Hamza Ahmed
2024-01-05 23:54:51 +00:00
parent c916352195
commit cf68a1d2cc
10 changed files with 745 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/posts/building-a-docker-swarm-with-pis.md
+203 -198
View File
@@ -3,229 +3,218 @@ title: Building a Docker Swarm With Pi's 👾 👾 👾 👾
date: 2021-12-12T17:23:32Z
author: "ahmza"
draft: false
weight:
weight:
tags: ["Homelab","Docker","Docker Swarm","Raspberry Pi","Pi4"]
tags: ["Homelab", "Docker", "Docker Swarm", "Raspberry Pi", "Pi4"]
description: "Pis are very capable computers, when banded together they're almost unstoppable"
canonicalURL: "https://blog.ahmza.com/Page%28/posts/building-a-docker-swarm-with-pis.md%29"
cover:
image: "/webp/posts/pi-docker-swarm/feature-anim.webp" # image path/url
alt: "Pi Docker Swarm" # alt text
caption: "<text>" # display caption under cover
relative: false # when using page bundles set this to true
image: "/webp/posts/pi-docker-swarm/feature-anim.webp" # image path/url
alt: "Pi Docker Swarm" # alt text
caption: "<text>" # display caption under cover
relative: false # when using page bundles set this to true
showtoc: true
searchHidden: false
---
> This post is no longer valid as the cluster runs Kubernetes. The build process
> is the same
## **Intro**
The _Raspberry Pi_ from it's first iteration has been a great hit.
Adopted by many as the computer of choice for projects, and for
very good reason. The Pi is a fully fledged computer that fits in
your pocket and the most recent edition has blown fans away with
the flagship model boasting an ARM64 CPU and 8GB of RAM.
Now one _Raspberry Pi 4 8GB_ is more than powerful enough for most
workloads and in some cases totally unnecessary, **so you may even
get away with a single 4GB model**. However taking into consideration
their relatively low price and power consumption why not go all out
and build a cluster.
The _Raspberry Pi_ from it's first iteration has been a great hit. Adopted by
most as the computer of choice for projects. The Pi is a fully fledged computer
that fits in your pocket and the most recent edition boasts an ARM64 CPU and 8GB
of RAM.
> **You can never have too many _Raspberry Pis_, just
not enough Ideas**
Now one _Raspberry Pi 4 8GB_ is more than powerful enough for most workloads,
**so you may even get away with a single 4GB model**. Taking into consideration
their low price and power consumption why not go all out and build a cluster.
My excuse to build a Pi cluster is, I would like to migrate the
services currently running on a **bare metal server that sucks
a lot of power to a low power solution.** What better way to do that
than a Pi cluster.
My excuse to build a Pi cluster: I would like to migrate the services running on
a **bare metal server that sucks a lot of power to a low power solution.**
### **My Setup**
* 4 x _Raspberry Pi 4 8GB_
* 4 x 32GB SD cards
* 4 x PoE hats
* 1 x 5 port PoE Switch
![Cluster](/img/pi-docker-swarm/cluster.webp)
- 4 x _Raspberry Pi 4 8GB_
- 4 x 32GB SD cards
- 4 x PoE hats
- 1 x 5 port PoE Switch
![Cluster(fix this image)](/img/pi-docker-swarm/cluster.webp)
### **Point of this post**
Hopefully after following this post you will have setup _Docker
swarm_ on a cluster of Pis **- stay even if you currently only have
one -** and understand how to bend it to your will 🦾.
### **Why** ***Docker Swarm*** 🧐
After following this post you will have setup _Docker swarm_ on a cluster of Pis
**- stay even if you have one -** and understand how to bend it
to your will 🦾.
### **Why** **_Docker Swarm_** 🧐
While we can manage each docker node - _Raspberry Pi_ - individually
, **_Docker swarm_ enables us to manage all the devices from a single
main node** making managing the cluster a lot easier.
Think of _Docker swarm_ as a set of 🤖 hands which will run tasks on
each individual node for us. These tasks can consist of:
* Cluster Management
* Deployment and Scaling across the cluster
* Multi-host networking
* load balancing
* etc...
each individual node for us. Some of these tasks are:
### **Well why not** ***Kubernetes***
Although _Kubernetes_ is awesome, **it adds a bunch of complexity that
is simply not needed in a home lab environment** where we only have
a small cluster. As well as that, from my experience creating
```docker-compose``` files is a lot easier than writing manifests.
**_Docker swarm_ can do everything I want it to and requires less
effort** which is great if you're like me 🥱 😂
- Cluster Management
- Deployment and Scaling across the cluster
- Multi-host networking
- load balancing
### **Well why not** **_Kubernetes_**
Although _Kubernetes_ is awesome, it adds a bunch of complexity I'm not ready
for. As well as that, from my experience creating _docker-compose_ files is a
lot easier than writing manifests. **In the future I will try _Kubernetes_**
### **Required Stuff**
> The links below are reccomendations/stuff I used
* [_Raspberry Pi_ 4B](https://thepihut.com/products/raspberry-pi-4-model-b?variant=31994565689406) (Reccomend 4-8 @ 8GB)
* [_Raspberry Pi_ PoE Hats](https://thepihut.com/products/power-over-ethernet-poe-hat-d-for-raspberry-pi) (If you can find the [PoE+](https://thepihut.com/products/raspberry-pi-poe-plus-hat) version get them)
* [Cluster Case](https://thepihut.com/products/cluster-case-for-raspberry-pi) (Get any case you want)
* [PoE switch](https://www.amazon.co.uk/gp/product/B076982FVC/) (Get one that has enough ports for the nuber of pis you want)
* [RJ45 Ethernet Cable](https://www.amazon.co.uk/gp/product/B00C4U030G) (Reccomend short cables as it will look neater)
* [Micro SD cards](https://www.amazon.co.uk/gp/product/B08J4HJ98L) (Get enough cards for the number of Pis, Get the biggest cards you can afford)
**This is without a doubt an expensive project**, however in my eyes
it's the cost of another server 😂 and the best part about this
project is **you can add to the cluster over time** spreding the cost
which is how I managed to purchase all the hardware. A suggestion
would be to **buy equipment that allows you room to grow** so an 8 port
PoE switch instead of a 4 port if you know you will scale to 8
nodes
- [_Raspberry Pi_ 4B](https://thepihut.com/products/raspberry-pi-4-model-b?variant=31994565689406) (Reccomend 4-8 @ 8GB) (Or newer)
- [_Raspberry Pi_ PoE Hats](https://thepihut.com/products/power-over-ethernet-poe-hat-d-for-raspberry-pi) ([PoE+](https://thepihut.com/products/raspberry-pi-poe-plus-hat)) (Check compatibility with Pi Version)
- [Cluster Case](https://thepihut.com/products/cluster-case-for-raspberry-pi) (Get any case you want)
- [PoE switch](https://www.amazon.co.uk/dp/B0769C24T1/) (get one with enough 1Gb ports for the number of pis you want)
- [RJ45 Ethernet Cable](https://www.amazon.co.uk/gp/product/B00C4U030G) (Recommend short cables as it will look neater)
- [Micro SD cards](https://www.amazon.co.uk/gp/product/B08J4HJ98L) (Get enough cards for the number of Pis, Get the biggest cards you can afford)
If you build this then you can safely say another server is not
needed. This cluster should be more than capable to serve all your
home needs even if you only have 2 nodes. Put it this way, if you
have 4 8GB nodes then you have a combined total of:
* 32GB of RAM
* 16 physical CPU cores @ 1.5GHz
**This is an expensive project**, but the best part is **you can add to the
cluster over time** spreading the cost, which is how I managed to buy all the
hardware. If I could start again though I would **buy equipment that allows room
to grow** so an 8 port PoE switch instead of a 4 port if you know you will scale
to 8 nodes
This cluster should be capable of serving most of your needs even with 2 nodes.
Put it this way, if you have 4 x 8GB nodes then you have a combined total of:
- 32GB of RAM
- 16 physical CPU cores @ 1.5GHz (You can OverClock !!)
#### **Money Saving Tips**
* Just get 1-2 nodes to start
* cut PoE and use normal power cables
* Use Raspberry Pis with less memmory eg 4GB
* Grab a really cheap case or make one using [these](https://www.amazon.co.uk/dp/B091KJNZLB/)
* Use smaller SD Cards try not to go below 32GB (Reccomended 64GB If money saved elsewhere)
- 1-2 nodes to start
- Drop PoE and use normal power cables
- Raspberry Pis with less memory i.e 4GB
- Cheaper case or make one using [these](https://www.amazon.co.uk/dp/B091KJNZLB/)
- Smaller SD Cards try not to go below 32GB (Recommended 64GB If money saved elsewhere)
## **BUILD TIME !!**
Pull yourself up by your bootstraps were jumping into this one
runnig 💨
### **Image the SD Cards**
So interesting fact the CPUs used in the raspberry pi 4 use an ARM64
archetecture. When looking for images of Raspbian you will only find
32bit variants which will work however will not make use of all the
hardware which is just a waste. 64bit images exist however are well
hidden, Dont fear 😨 **you can download the latest images from the
[raspberrypi.org index](https://downloads.raspberrypi.org/raspios_arm64/images/)**
To write the cards I use the [**Raspberry Pi imager**](https://www.raspberrypi.com/software/).
once you have **rpi-imager** installed and you have the 64bit raspbian
image, open ```sudo rpi-imager``` and on the "CHOOSE OS" section go to
the bottom and select use custom image.
The CPU in a RaspberryPi 4 is the ARM64 architecture. When selecting Raspbian
pick the 64-BIT option. 32bit variants will work but not make use of the hardware
capabilities.[RaspberryPi Imager](https://www.raspberrypi.com/software/)
> We dont need to unzip the file as the tool is clever enough to
handle that for us
when prompted with **"Use OS Customisation"** go with **"Edit Settings"**.
General:
**Before removing the SD card, mount the boot partition - usually the
first partition, look for a file called cmdline.txt - and create an
empty file called ssh** ```sudo touch ssh``` this basically enables
SSH on the Pis first boot allowing us to access the Pis in a totally
headless manner.
- Set the hostname
- Set a username and password
- Set Locale settings
Services:
- Enable SSH
- Allow public-key authentication only
- Set your SSH Key
> If you dont have an SSH key, read my post on SSH [Securing
> SSH](https://blog.ahmza.com/posts/securing-ssh/)
### Assemble the cluster
So assembly is very dependant on the kit you bought, and more
often than not the stuff needed here comes with instructions or is
self explanitory. So go ahead and assemble the cluster 🛠
**There is an order of operations that I reccomend:**
* Install PoE hats onto the Pis
* Install the SD Cards into the Pis
* Assemble the case and install Pis
* plug in ethernet cables
Assembly depends on the case you bought, instructions should be included. Go
ahead and piece together the cluster 🛠
**Now allow the Pis to boot for a few minutes so it can resize the
**There is an order of operations that I recommend:**
- Install PoE hats onto the Pis
- Install the SD Cards into the Pis
- Assemble the case and install Pis
- plug ethernet cables in
**Now allow the Pis to boot for a few minutes**
### **Sort out the Network bits**
This is the easy part, there are two methods you can use to set a
static IP address, one is on your DHCP server and the other is on
the device itself. Personally I prefer to set static IPs on the
device itself so I will cover that here.
To find the current ip of your Pis you can go to your DHCP server
and look for the DHCP leases for the Pis.
There are two methods you can use to set a static IP address, one is on your
DHCP server and the other is on the device itself. Personally I prefer to set
static IPs on the device itself so I will cover that here.
To find the current ip of your Pis you can go to your DHCP server and look for
the DHCP leases for the Pis.
> there are naming conventions for naming primary and secondary
things but this is your cluster so be creative with it, I have my
primary node named ```Goob``` and the secondary nodes named ```Doris1```
```Doris2``` and ```Doris3```
[reference](https://www.youtube.com/watch?v=oFb7FVe8sdQ)
> things but this is your cluster so be creative with it, I have my
> primary node named `Goob` and the secondary nodes named `Doris1` > `Doris2` and `Doris3` > [reference](https://www.youtube.com/watch?v=oFb7FVe8sdQ)
**SSH to the Pi** then open and edit the following files
* /etc/hosts
* edit this line with your hostname: ```127.0.1.1 goob```
* /etc/hostname
* edit this file with your hostname: ```goob```
* /etc/dhcpcd.conf
* knowing we will be using only static IPs here I delete
everything in the file add the following
* ```
interface eth0
static ip_address=10.10.10.20/24
static routers=10.10.10.1
static domain_name_servers=10.10.10.1
```
**Now reboot the pi** ```sudo shutdown -r now```
**Once the Pis are back up log into each one and change the default
passwords** you may also want to create a new user for yourself but
thats totally optional.
- /etc/dhcpcd.conf
\_ knowing we will be using only static IPs here I delete
everything in the file add the following
```
interface eth0
static ip_address=<your-Pi-ip>/24
static routers=<your-router-ip>
static domain_name_servers=<your-dns-ip-usually-same-as-router>
```
### **Provision with** ***Ansible***
you want to **create an _Ansible_ control node** and that is as simple as
installing a package onto your machine. _Ansible_ is built using python
which makes it easy to install as **we can use the python pip package
manger.** To check you have it installed do ```pip --version``` if
it's not installed do the following
**Now reboot the pi**
```bash
sudo shutdown -r now
```
```curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py```
### **Provision with** **_Ansible_**
```python get-pip.py --user```
Ansible will allow us to setup our RaspberryPi Cluster from a remote machine
where we only administer the commands once and Ansible will replicate across the
cluster.
**once you have pip installed proceed to installing _Ansible_**
> Make sure pipx is installed. [Install
> Instructions](https://pipx.pypa.io/stable/)
[Docs](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html)
Install Ansible
[Install Docs](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible-with-pipx)
```pip install ansible```
```bash
pipx install --include-deps ansible
```
Now that _Ansible_ is installed we can create a runbok that defines the
steps _Ansible_ needs to take on the remote machine to correctly
provision it. This playbook is written in yaml so get ready to make
sure your indents are correct 😬
A playbook defines steps _Ansible_ needs to take on the remote machine to
provision it. Playbooks are written in yaml so get ready to make sure your
indents are correct 😬
**Creating the playbook**
now you can copy what I have and it should work - after changing a
thing or two - which is fine however I encourage you to copy what I
have, read the [Docs](https://docs.ansible.com/ansible/latest/user_guide/index.html)
and make any changes you feel are necessary.
Copy what I have and it should work -- after changing a thing or two -- I
encourage you to read the Ansible
[Docs](https://docs.ansible.com/ansible/latest/user_guide/index.html) and make
any changes you feel are necessary.
> Check the [git repo](https://gitlab.com/AHMZA/pi-docker-swarm-ansible-deploy/-/tree/main) for the most up to date yaml file im using.
> Check this [Gitlab Repo](https://gitlab.com/ahmza/ansible-playbooks/-/tree/main/docker-swarm?ref_type=heads) for the most up to date playbook.
**This script assumes you are using SSH keys, which you should but if
not you can skip the whole "Configure SSH" block,**
***I RECCOMEND YOU USE SSH KEYS*** - [read this](https://blog.ahmza.com/posts/securing-ssh/)
**_I RECCOMEND YOU USE SSH KEYS_** - [read this](https://blog.ahmza.com/posts/securing-ssh/)
**Create the deploy.yml**
```bash
# Open the yaml file in a text editor
vim deploy.yml
```
```yaml
---
- hosts: piCluster
become: yes
tasks:
- name: Configure SSH
block:
- name: Auth local SSH Keys
ansible.posix.authorized_key:
user: <pi_user>
@@ -239,7 +228,7 @@ not you can skip the whole "Configure SSH" block,**
copy:
src: ./pi_rsa
dest: /home/<pi_user>/.ssh/id_rsa
mode: '0600'
mode: "0600"
owner: <pi_user>
group: <pi_user>
@@ -247,11 +236,10 @@ not you can skip the whole "Configure SSH" block,**
copy:
src: ./pi_rsa.pub
dest: /home/<pi_user>/.ssh/id_rsa.pub
mode: '0644'
mode: "0644"
owner: <pi_user>
group: <pi_user>
- name: Check for any updates and install them
apt:
update_cache: true
@@ -266,14 +254,12 @@ not you can skip the whole "Configure SSH" block,**
- vim
- curl
- name: Check Docker is installed
shell: "docker --version"
register: docker_version
- name: Install Docker
block:
- name: Download Docker script
command: curl -fsSL https://get.docker.com -o get-docker.sh
@@ -297,7 +283,6 @@ not you can skip the whole "Configure SSH" block,**
when: docker_version.stdout.find('Docker version')
- name: check Docker compose is installed
shell: "docker-compose --version"
register: dc_version
@@ -306,7 +291,6 @@ not you can skip the whole "Configure SSH" block,**
command: sudo pip3 -v install docker-compose
when: dc_version.stdout.find('docker-compose version')
- name: wait for network on boot
command: raspi-config nonint do_boot_wait 0
@@ -315,49 +299,57 @@ not you can skip the whole "Configure SSH" block,**
```
The steps this playbook takes are as follows:
1. Copy your SSH keys to the pi authorized_keys
2. Copy same Pi ssh key between the Pis for intercommunication
3. Check for and Install updates
4. Install any build dependancies
4. Install any build dependencies
5. Check if Docker is Installed
* Download the Docker Script
* Install Docker from script
* Enable the Docker service
* Add user to Docker group
- Download the Docker Script
- Install Docker from script
- Enable the Docker service
- Add user to Docker group
6. Check if docker-compose is installed
* Install docker-compose (This will take a while on each pi so be patient, grab a coffee)
9. Change raspi config to wait for network before boot
10. Reboot
- Install docker-compose (This will take a while on each pi so be patient, grab a coffee)
7. Change raspi config to wait for network before boot
8. Reboot
take some time to read through the playbook, read the docs and try to
understand whats going on. Change the usernames to your username.
**Make the hosts file in the same directory as the deploy.yml**
``` yaml
[piCluster]
10.10.10.20
10.10.10.21
10.10.10.22
**Create an inventory hosts file in the same directory as the deploy.yml**
```bash
# Open file in a text editor
vim hosts
```
* `[piCluster]` - specifies a group of hosts
* `10.10.10.xx` - specifies the Pis IP
```yaml
[piCluster]
<ip address 1>
<ip address 2>
<ip address 3>
```
- **_[piCluster]_** - specifies a group of hosts
- **_<ip address>_** - change these to the Pis IP addresses
**create the Pi Intercommunication keys**
``` bash
```bash
ssh-keygen
# Please note the ".pi_rsa" here
Enter file in which to save the key (/home/ahmza/.ssh/id_rsa): .pi_rsa
```
* save the keys in the same directory the deploy.yml file is
**heres the command to run the playbook.**
`ansible-playbook -i hosts -kK deploy.yml -c paramiko`
**_ansible-playbook -i hosts -kK deploy.yml_**
* `ansible-playbook` - the command to run the playbook
* `-i hosts` - specifies the host/inventory file
* `-kK` - asks you for password for the SSH connection(-k) and the sudo password(-K)
* `deploy.yml` - the _Ansible_ playbook
* `-c paramiko` - Use the paramiko connection method, I found this works best with ssh keys
- _ansible-playbook_ - the command to run the playbook
- _-i hosts_ - specifies the host/inventory file
- _-kK_ - asks you for password for the SSH connection(-k) and the sudo password(-K)
- _deploy.yml_ - the _Ansible_ playbook
You can do so much with _Ansible_, for example mount network shares etc,
if you can do it in the cli you can do it in _Ansible_ !!
@@ -366,35 +358,35 @@ Now that the _Ansible_ script is done we can go ahead and actually
configure all the Pis into the cluster.
### **Creating/Joining the swarm**
Creating and joining the cluster is easy - It is manual and tedious so
this is something you could impliment into the ansible playbook using
multiple host groups.
Creating and joining the cluster is easy - It is manual and tedious so this is
something you could implement into the _Ansible_ playbook using host groups.
**Follow these steps to create the swarm:**
1. ssh to the main Pi, this will be the manager node.
2. run `docker swarm init`
* This will return a command and token. Make sure to copy them
- This will return a command and token. Make sure to copy them
3. exit the main Pi
4. ssh to one of the worker nodes
5. paste and run the copied command
6. do the same on the remaining worker nodes
You should now have a correctly configured cluster !!
You should now have a configured cluster !!
### **Optional Step - Portainer**
Portainer is a webgui that will allow us to manage the cluster, it will
### **Optional Step - _Portainer_**
_Portainer_ is a webgui that will allow us to manage the cluster, it will
make deploying and destroying containers across the cluster a breeze.
I do understand that all of what portainer does can simply be accomplished
in the command line, however portainer just makes it less hastle to manage.
**Install Portainer**
**Install _Portainer_**
Firstly we want to download the yaml file that will be used to deploy
the portainer contiainer on our cluster.
the _Portainer_ container on our cluster.
`curl -L https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml`
Now we can actually deploy portainer to the cluster using the following
Now we can actually deploy _Portainer_ to the cluster using the following
command:
`docker stack deploy -c portainer-agent-stack.yml portainer`
@@ -412,10 +404,10 @@ to create a private repository to host any images that you need to run
across the cluster.
to create this private repository create a file called docker-compose.yml
in a folder called repository or whatever and enter the following.
``` yaml
in a folder new folder called repository and enter the following.
version: '3.9'
```yaml
version: "3.9"
services:
registry:
image: registry:2
@@ -428,26 +420,39 @@ services:
**Be sure to change pi-user to your pi user**
**create a file /etc/docker/daemon.json and eter the following**
``` json
```bash
vim /etc/docker/daemon.json
```
```json
{
"insecure-registries": ["localhost:5000", "<pi-address>:5000"]
"insecure-registries": ["localhost:5000", "<pi-address>:5000"]
}
```
where `pi-address` is the ip address of the main swarm node.
**restart the docker service with `sudo systemctl restart docker`**
where **_pi-address_** is the ip address of the main swarm node.
\*\*restart the docker service
```bash
sudo systemctl restart docker
```
now we need to deploy the registry to the stack like we did with portainer
`docker stack deploy -c docker-compose.yml registry`
```bash
docker stack deploy -c docker-compose.yml registry
```
and we should have a private registry setup that we can use to push our
custom docker images to.
## **Conclusion**
Well done, you have successfully setup a docker swarm cluster on a bunch
of Pis and along the way you managed to learn a bit about SSH, Ansible,
docker-compose, yaml and networking on Pis. The next step would be to
setup a few containers but I'll leave that to you, dont be affraid to
play around and even ruin the cluster, you can always rebuild it so have
play around and even ruin the cluster, you can always rebuild it have
fun 😊