DevOps, Productivity, Career // etogeek blog

Maintenance mode in Nginx

Mon, 27 Oct 2025 00:00:00 +0000

This post is also available in Russian.

What is it?
#

There is no built-in “mode” like this in Nginx. I just use this name for the ability to manually switch the site to serve a specific page instead of the normal content.

Right now we can serve a specific HTML page instead of the default error page.

For example, instead of the default 504 error we can return a nice page with the company logo and a clear polite message like “The service is temporarily unavailable. We’re aware of the problem; please try again in a minute.”

error_page 500 501 503 502 504 /static/errors/5xx.html

What if we know there will be maintenance today from 20:00 and the service will be down? We turn on maintenance mode with information about the work and an ETA.

Second case: the service crashed hard, engineers are handling the incident, and users are seeing 500s, 503s, 504s, and timeouts. Turn on maintenance mode so everyone sees a stable message.

A not-so-good attempt
#

The first solution you’ll find online: use if to check for a file on the server (for example /etc/nginx/maintenance.flag). If the file exists — return the page; if not — continue with the standard Nginx config. Something like this:

server {
  	....
  	error_page 503 /maintenance.html;
	
  	location = /maintenance.html {
  	    root /etc/nginx/maintenance/;
  	    internal;
  	}
	
  	location / {
  	  if (-f /etc/nginx/maintenance.flag) {
  	     return 503;
  	  }
  	  proxy_pass http://127.0.0.1:3000;
  	}
}

This works, but here the file existence is checked on EVERY request. That can be wasteful. It looked fine in the dev environment, but caused slowdowns in production.

Why `return 503`?
#

Why should we return HTTP 503 even though we’re sending a normal, working HTML page?

503 means the service is temporarily unavailable (maintenance or overload). More details.

Search engines do not index pages that return 503. Imagine Google’s crawler hits your site during downtime and stores “Under maintenance” as the page content.

Also, browsers should not cache such pages, so after maintenance ends the user doesn’t keep seeing the maintenance page from cache.

How to do it with a config include?
#

The idea:

we have a file maintenance_on.conf with a single line: return 503;
when needed, include this file in the desired locations via include maintenance_on.conf
gracefully restart Nginx (reload)

Implementation details:

Create a directory to keep things clean: mkdir /etc/nginx/maintenance
Put your maintenance.html page there. Vibe-code it…
Create the config file maintenance_on.conf:

return 503;

Create an empty config file maintenance_off.conf
Create a symlink maintenance_current.conf

ln -sfn /etc/nginx/maintenance/maintenance_off.conf /etc/nginx/maintenance/maintenance_current.conf

Add a few lines to the Nginx server block:

server {
		...
    error_page 503 /maintenance.html;

    location = /maintenance.html {
        internal;
        root /etc/nginx/maintenance/maintenance.html;
        add_header Cache-Control "no-store" always;
        add_header Retry-After "3600" always;
    }

    location / {
        include /etc/nginx/maintenance/maintenance_current.conf;
        ...
    }
}

What happens here:

We tell Nginx to serve /maintenance.html on a 503 error.
We configure the /maintenance.html location that serves our service page. internal ensures the client cannot directly request /maintenance.html or any other internal file.
We add headers so the browser does not cache the page, plus Retry-After.
In the main (or any) location we include our symlink.

So to enable maintenance mode we only need to switch the symlink and gracefully reload Nginx:

ln -sfn /etc/nginx/maintenance/maintenance_on.conf /etc/nginx/maintenance/maintenance_current.conf
nginx -t && systemctl reload nginx

To disable it, point the symlink back to the _off file and reload Nginx again.

How to automate it — up to you.

How to Install GitLab Runner in Docker Swarm

Sun, 18 Aug 2024 10:06:32 +0000

This post is also available in Russian.

In my project, I have a Docker Swarm cluster where Jenkins Agents are dynamically launched to execute all the jobs from Jenkins. Why not in Kubernetes? Because maintaining a Kubernetes cluster just for jobs is not optimal; Swarm is much simpler to manage.

Recently, I set up a Self-Hosted GitLab, but I couldn’t find an official guide on how to launch a GitLab Runner in a Swarm cluster with autoscaling – meaning that the runner would automatically spin up additional containers on Swarm nodes. I had to gather the information bit by bit. Here’s the method I used.

The guide consists of three steps:

Obtain the registration token
Distribute the runner configurations to the machines
Deploy the runners

Obtaining the Runner Authentication Token
#

Previously, a runner registration token was used to register runners, but it has now been deprecated. GitLab has started using Runner Authentication Token for connecting runners.

Go to the project, group, or instance settings to register a new Runner: Admin Settings -> CI/CD -> Runners -> New instance runner:

Specify the Tags. Later, we’ll use these tags in our GitLab CI pipelines to ensure they run on specific runners. For example, I use the tag swarm.

On the next page, copy the token and save it somewhere:

Now, GitLab will be waiting for a runner to register with this token.

Configuring the Runners
#

We can launch a container with the GitLab Runner, exec to it, and execute the register command. During registration, GitLab Runner will save the token and minimal settings in the /etc/gitlab/config.toml file inside the container. But instead of manually executing a register command on each container on every machine, let’s create a configuration template and distribute it to the Swarm nodes.

Prepare the configuration file:

concurrent = 5 # Concurrent containers per node
check_interval = 0

[[runners]]
  name = "docker-swarm-runner"
  url = "https://<YOUR-GITLAB-INSTANCE>/"
  token = "<GITLAB AUTHENTICATION TOKEN>"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"  # Default image for jobs
    privileged = true  # If we need Docker-in-Docker
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    allowed_pull_policies = ["always", "if-not-present", "never"]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Replace the token with the one you obtained in the previous step. Also, specify the address of your GitLab instance.

Now, on each Swarm node, you need to create a directory for the configuration and place this file there. How you do this is up to you, but I use Ansible.

Remember, storing the token in a Git repository is unsafe. It’s bad practice. In the case of Ansible, you can use Ansible Vault or third-party solutions like Hashicorp Vault or Infisical.

Copying the Configuration File to Nodes
#

Here’s an example of Ansible code to copy the configuration file:

templates/config.toml.j2:

concurrent = 5
check_interval = 0

[[runners]]
  name = "docker-swarm-runner"
  url = "{{ gitlab_instance_url }}"
  token = "{{ gitlab_runner_swarm_auth_token.value }}"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"  # Default image for jobs, can be overridden in .gitlab-ci.yml
    privileged = true  # Enable if you need to run Docker-in-Docker or privileged containers
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    allowed_pull_policies = ["always", "if-not-present", "never"]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

tasks/main.yml:

- name: Create directories
  ansible.builtin.file:
    path: /etc/gitlab-runner/config
    state: directory
    recurse: true

- name: Copy config file
  ansible.builtin.template:
    src: config.toml.j2
    dest: /etc/gitlab-runner/config/config.toml

The configuration file should be on all Swarm cluster nodes.

Deploying GitLab Runner in Swarm
#

Swarm works with the familiar docker-compose files, so let’s create one. I also use Ansible for copying, so here’s an example jinja template:

services:
  gitlab-runner:
    image: gitlab/gitlab-runner:{{ gitlab_runner_image_version }}
    deploy:
      mode: global
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/gitlab-runner/config:/etc/gitlab-runner
    networks:
      - gitlab-net

networks:
  gitlab-net:
    external: false

We specify deploy mode: global, so the container is launched on all cluster nodes. This is like a DaemonSet in Kubernetes.
We connect the directory with the config.toml file.
Additionally, we mount the Docker socket so that the GitLab runner can manage our Docker engine and create new containers (this is necessary for autoscaling).
We specify the version of the GitLab Runner container; you can find available versions on Docker Hub.

I copy this file using Ansible:

- name: Copy compose file
  ansible.builtin.template:
    src: docker-compose.yml.j2
    dest: /etc/gitlab-runner/docker-compose.yml

You need to run the Swarm stack from the manager node (the main cluster node), but the file will be on all nodes. It’s redundant, but it’s easier for me this way, so I don’t have to add conditions in Ansible.

Go to the manager node and execute:

docker stack deploy -c docker-compose.yml gitlab-runner

After a minute, check:

root@swarm-node01:/home/ysemyenkov# docker stack ps gitlab-runner
ID             NAME                                                    IMAGE                                 NODE           DESIRED STATE   CURRENT STATE        ERROR     PORTS
pnkxq7yf3wm3   gitlab-runner.1tlong-name   gitlab/gitlab-runner:ubuntu-v17.3.0   swarm-node01   Running         Running 2 days ago
ih4822qch9ew   gitlab-runner.3tlong-name   gitlab/gitlab-runner:ubuntu-v17.3.0   swarm-node03   Running         Running 2 days ago
xzj5yt1cv5ae   gitlab-runner.2tlong-name   gitlab/gitlab-runner:ubuntu-v17.3.0   swarm-node02   Running         Running 2 days ago

If everything is okay, you’ll see the registered Runner in GitLab:

Testing
#

We can create a test pipeline that runs a dozen jobs simultaneously:

stages:
  - test

parallel-jobs:
  stage: test
  tags:
    - swarm
  script:
    - echo "Starting sleep for 600 seconds"
    - sleep 600
  parallel:
    matrix:
      - JOB_NAME: "job1"
      - JOB_NAME: "job2"
      - JOB_NAME: "job3"
      - JOB_NAME: "job4"
      - JOB_NAME: "job5"
      - JOB_NAME: "job6"
      - JOB_NAME: "job7"
      - JOB_NAME: "job8"
      - JOB_NAME: "job9"
      - JOB_NAME: "job10"

Run the pipeline and see that the containers launched in the required number and on different nodes. You can view it with docker ps command on the nodes or through monitoring tools. For example, I prefer viewing cAdvisor metrics in Grafana:

You can see that the top three containers are the main GitLab Runner processes, which receive tasks and create dynamic containers for jobs. Below are our containers with sleep jobs running simultaneously on different nodes.

That’s it.

Difference: Authentication and Authorization

Thu, 18 Apr 2024 00:00:00 +0000

Authentication and authorization — similar words with different meanings, and they are easy to confuse. Let’s explore the differences.

Authentication
#

Authentication is a process that verifies you are who you claim to be. An example of authentication is entering a username and password. The system checks these credentials to confirm your identity.

In Linux systems, authentication is handled by files such as:

/etc/passwd — stores user information.
/etc/shadow — contains encrypted user passwords.
/etc/nsswitch.conf — directs where to look for user and group information and other resources, including DNS or network file system services. It tells the system to look for data in local files or, for example, in LDAP.

Authorization
#

Authorization follows authentication and determines your access to resources or actions. It checks whether you are permitted access to certain service areas.

In Linux, access rights are defined through:

/etc/group — describes user groups and their members.
/etc/sudoers — specifies who can execute commands with superuser privileges.

Real-life Example
#

When you show your passport at the border, that’s authentication — you prove that you are John Smith. Deciding whether you can leave the country is a matter of authorization.

Important to remember: Authentication always precedes authorization. The system first needs to know who you are and then decides what you are allowed to do.

Прокомментировать статью, поделиться идеями, поболтать и задать вопрос можно в 👉 телеграм-чате, а так же обязательно подписаться на 👉 телеграм-канал!

How to commit only part of a file?

Thu, 08 Feb 2024 00:00:00 +0000

TL;DR;
#

git add -p

A bit more detail
#

The git add -p (or --patch) command goes through all files and asks about each change, what you want to do with it:

❯ git add -p
diff --git a/ansible/file.yml b/ansible/file.yml
index 73f6b4d..899610f 100644
--- a/ansible/file.yml
+++ b/ansible/file.yml
@@ -52,7 +52,8 @@
       loop:
-        - "service"
+        - "node_modules"
+        - "systemd_templates"
       
(1/1) Stage this hunk [y,n,q,a,d,e,?]?

Then it’s an interactive input. Well, the most important are “y” and “n" — to add these changes to the index or skip them. There are options to skip or add “to the end of the file”. It can also be used as git add -p filename.

Reminder, the index in git is a list of files that will be committed.

Importantly, this action adds our change only to the index. It still needs to be committed.

How else can it be used?
#

As I wrote here, I always used either git add . or git add <filename>. But there’s a catch:

with git add ., extra files about which you might have forgotten can get into the index, and then into the commit

Here, git add -p allows you to more consciously (thoughtfully) go through the files before adding them to the index.

Also, a reminder that to remove a file from the index, you can use the command git reset filename.

What do I listen to when I work

Wed, 07 Feb 2024 00:00:00 +0000

What I Listen On?
#

AirPods Pro 2 — powerful noise cancellation system, fit well in the ears, still hold a charge well, switches between devices correctly. They are over a year old, I use them every day. I’m deeply integrated into the Apple ecosystem and don’t think I’ll switch to another headphone brand in the foreseeable future.

What Do I Listen To?
#

It really depends on my mood, the task at hand, energy level, but here are the main playlists from my bookmarks:

Discovery of the year — hip-hop is great for work, made such a playlist on Spotify.
If I want something a bit more calming, I turn on lo-fi girl. Either on Spotify, or Youtube.
I always find it hilarious that the soundtrack from the game Stronghold Crusader tops my Spotify Wrapped for the second year in a row, but that doesn’t take away from the fact that it’s a really cool 40-minute playlist. Spotify, Youtube.
A very long playlist that is both relaxing and keeps you focused with an ambient remix of the soundtrack to the movie Dune.
Another long playlist, repetitive but more relaxing — music from the game Witcher 3. Youtube.
When in a cheerier mood, I might play my retro playlist - Spotify.
Two playlists about 15-20 minutes each — two radio stations from the game GTA 3 on Youtube: Game Radio, and Head Radio.
Quite upbeat music, can be distracting, suited for less focused and routine work — 40 minutes, album by the band The Medics (NL) Spotify, Youtube.

Difference between $VAR and ${VAR}?

Sun, 28 Jan 2024 00:00:00 +0000

This is really a frequent question. I’m explaining in addition to the post(sorry, it is not translated yet) about variable substitution in a file.

There’s no rocket science here, and the difference only lies in how the command interpreter (Bash, zsh, sh, thousands of them) determines the boundaries of the variable name:

$VAR — works well when the variable name is separated from other text by unused symbols, such as a space, or a backslash. But if we insert the variable into a word, it will be interpreted incorrectly:

VAR=eto
echo $VAR      # Will output eto
echo $VARgeek  # Will output the value of VARgeek, if it exists, otherwise — empty

${VAR} — more reliable. Helps to accurately separate the variable name from other text:

VAR=eto
echo ${VAR}     # Will output eto
echo ${VAR}geek # Will output etogeek

DevOps, Productivity, Career // etogeek blog

Maintenance mode in Nginx

What is it? #

A not-so-good attempt #

Why return 503? #

How to do it with a config include? #

How to Install GitLab Runner in Docker Swarm

Obtaining the Runner Authentication Token #

Configuring the Runners #

Copying the Configuration File to Nodes #

Deploying GitLab Runner in Swarm #

Testing #