How to renew Rancher certificates when expired

I installed a rancher server to manage my own kubernestes cluster one year ago. I used Default Rancher-generated Self-signed Certificate mode for fast deploying.

docker run -d --restart=unless-stopped \
  -p 80:80 -p 443:443 \
  --privileged \
  rancher/rancher:latest

My cluster works well for year. And last weeks, i can’t access to Rancher UI. When i checked rancher container log, i got errors

“ x509: certificate has expired or is not yet valid “

Alter google to find a solution, i found this topic https://github.com/rancher/rancher/issues/26984. @justincarter and many peoples got same problem with me.

I tried to follow the recomendation of @dnauck in this topic but certificate of rancher server was not changed.

sudo timedatectl set-ntp off
sudo date --set="2021-03-30 09:03:00.000" (date before expiration)
sudo docker exec -it rancher sh -c "rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json"
kubectl delete secret -n kube-system k3s-serving (from the rancher cluster manager ui via Kubectl button)
sudo timedatectl set-ntp on
sudo docker restart rancher

So i decided to update rancher certificate with my own created certificate.

There is what i did with my system:

Step 1 — Installing Easy-RSA

Login to server which you choose to work as CA Server with the non-root sudo user that you created during the initial setup steps and run the following:

sudo apt update
sudo apt install easy-rsa

Step 2 — Preparing a Public Key Infrastructure Directory

Create new working directory.

Note: Make sure that you do not use sudo to run any of the following commands, since your normal user should manage and interact with the CA without elevated privileges.

mkdir ~/easy-rsa
chmod 700 ~/easy-rsa

This will create a new directory called easy-rsa in your home folder. We’ll use this directory to create symbolic links pointing to the easy-rsa package files that we’ve installed in the previous step. These files are located in the /usr/share/easy-rsa folder on the CA Server.

Create the symlinks with the ln command:

ln -s /usr/share/easy-rsa/* ~/easy-rsa/

Finally, initialize the PKI inside the easy-rsa directory:

cd ~/easy-rsa
./easyrsa init-pki

Output

init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /home/{your-user}/easy-rsa/pki

After completing this section you have a directory that contains all the files that are needed to create a Certificate Authority. In the next section you will create the private key and public certificate for your CA.

Step 3 — Creating a Certificate Authority

Before you can create your CA’s private key and certificate, you need to create and populate a file called vars with some default values. First you will cd into the easy-rsa directory, then you will create and edit the vars file with nano or your preferred text editor:

cd ~/easy-rsa
nano vars

Once the file is opened, paste in the following lines and edit each highlighted value to reflect your own organization info. The important part here is to ensure that you do not leave any of the values blank:

~/easy-rsa/vars
set_var EASYRSA_REQ_COUNTRY    "VN"
set_var EASYRSA_REQ_PROVINCE   "HANOI"
set_var EASYRSA_REQ_CITY       "City of Peace"
set_var EASYRSA_REQ_ORG        "DPBDCORP"
set_var EASYRSA_REQ_EMAIL      "admin@example.com"
set_var EASYRSA_REQ_OU         "Community"
set_var EASYRSA_ALGO           "ec"
set_var EASYRSA_DIGEST         "sha512"

When you are finished, save and close the file. If you are using nano, you can do so by pressing CTRL+X, then Y and ENTER to confirm. You are now ready to build your CA.

To create the root public and private key pair for your Certificate Authority, run the ./easy-rsa command again, this time with the build-ca option:

./easyrsa --batch build-ca nopass

Note: Rancher requires certificates that it must include subjectAltName support. So we add them to SSL v3 certifcates.

## Configure valid time in 10 years
export EASYRSA_CERT_EXPIRE=3650
./easyrsa --batch --req-cn=example.com gen-req example.com nopass
./easyrsa --batch --subject-alt-name="DNS:www.example.com,DNS:*.example.com,IP: 172.16.183.154", IP: 127.0.0.1" sign-req server example.com

Verify cert content:

openssl x509 -in pki/issued/example.crt -noout -text

Copy certificate, key and ca files to rancher server and rename to match with rancher requirements

pki/issued/example.com.crt  -> /etc/rancher/ssl/cert.pem
pki/issued/example.com.key  -> /etc/rancher/ssl/key.pem
pki/issued/ca.crt  -> /etc/rancher/ssl/cacerts.pem

Backup Rancher container

Use this guild to backup container data and create new volume for data:

https://rancher.com/docs/rancher/v2.x/en/backups/v2.5/docker-installs/docker-backups/

Run rancher server:

docker run -drestart=unless-stopped -p 80:80 -p 443:443 --volumes-from rancher-backup -privileged -v /etc/rancher/ssl/cacerts.pem:/etc/rancher/ssl/cacerts.pem -v /etc/rancher/ssl/cert.pem:/etc/rancher/ssl/cert.pem -v /etc/rancher/ssl/key.pem:/etc/rancher/ssl/key.pem rancher/rancher:stable rancher

Verify ssl connection to Rancher from client

openssl s_client -CAfile cacerts.pem -connect example.com:443
openssl s_client -CAfile cacerts.pem -connect 172.16.183.154:443

If response has verify code is 0, certificate rotation is successful

Timeout : 300 (sec)
Verify return code: 0 (ok )

Redeploy rancher-agent

Login into Rancher UI, you still got an error message: “Cluster unavailable” and can’t acccess to Kubernestes cluster. The reason is that rancher-agent is still connect to rancher server with old token and ca-checksum.

You can check rancher-agent logs to see errors

docker container ls | grep agent
docker logs {rancher-agent-container} --tail 100 

To connect agent to rancher server again, you stop current agents and start new instances

Login to Rancher UI:

Login -> Edit -> Customine Node Run Command

Copy command in text box and run on new nodes

sudo docker run -d — privileged — restart=unless-stopped — net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.5.7 — server https://172.16.183.154— token <token>— ca-checksum <check_sum> — etcd — controlplane — worker

Login into Rancher UI again, you’ll see cluster status is active.

References:

How To Set Up and Configure a Certificate Authority (CA) On Ubuntu 20.04 | DigitalOcean