DEVOPS PLAYBOOK · GITLAB CI/CD · DOCKER-IN-DOCKER

Mastering On-Premise CI/CD: Bootstrapping & Debugging GitLab Runners

When transitioning from managed cloud platforms (like GitHub Actions or GitLab SaaS) to a self-hosted, air-gapped, or on-premise infrastructure, engineers are abruptly introduced to a harsh reality: Code does not compile itself.

Recently, I was architecting a CI/CD pipeline on a private GitLab instance. After writing the .gitlab-ci.yml pipeline and pushing to the main branch, the pipeline immediately stalled, displaying a notorious warning:

"This job is stuck because you don't have any active runners that can run this job."

To process the workloads, I needed to bootstrap a dedicated Linux compute node, configure it for Docker-in-Docker (DinD) build executions, and connect it to the master GitLab server.

What seemed like a simple binary installation turned into a masterclass in DevOps networking and security token management. Here is how I architected the runner and bypassed the networking roadblocks I hit along the way.

Phase 1: Bootstrapping the Compute (The Privileged Executor)

To execute pipeline jobs, GitLab relies on a background service called a gitlab-runner. I SSH'd into my deployment Linux server and systematically pulled the binaries and registered the service:

Bash

# 1. Download the Linux binary
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

# 2. Grant execution permissions
sudo chmod +x /usr/local/bin/gitlab-runner

# 3. Create an isolated system user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# 4. Install and initialize the service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

The Registration Configuration:

To connect this runner to GitLab, I utilized the docker executor. Because our pipeline requires the container to spin up other containers (to build our application images), I passed the --docker-privileged flag, effectively granting the DinD environment root access to the host engine.

Bash

sudo gitlab-runner register \
  --non-interactive \
  --url "http://192.168.1.X/" \
  --registration-token "SECRET_TOKEN" \
  --executor "docker" \
  --docker-image "docker:24.0.5" \
  --description "docker-runner-deploy-machine" \
  --docker-privileged

While the setup architecture was sound, connecting on-premise components always surfaces network and security traps. Here are my Trench Notes from resolving them.

Trench Notes: Overcoming Self-Hosted Networking & Security

Roadblock 1: The "Ghost Token" (403 Forbidden)

The Incident

Upon running the gitlab-runner register command, the CLI immediately crashed and threw a PANIC state:

POST http://192.168.1.X/api/v4/runners: 403 Forbidden. PANIC: Failed to register the runner.

The Root Cause

A 403 Forbidden at this specific step is an AuthN rejection. When navigating the GitLab UI earlier to view runner configurations, I had accidentally refreshed the page near the "Reset Token" module. GitLab’s security daemon instantly invalidated the existing registration token. The terminal was aggressively sending a dead "ghost" token to the API.

The Fix

I pulled a fresh cryptographic token from the Web UI, inserted it into the CLI command, and the API successfully handshaked with my compute node.

Roadblock 2: The Container DNS Isolation Trap

The Incident

Once the runner attached, the CI/CD job successfully began initializing. However, the exact moment the isolated Docker container attempted to execute git clone to pull my repository, it crashed fatally:

fatal: http://gitlab.internal.com/... not valid: is this a git repository?
ERROR: Job failed: exit code 1

The Root Cause

This is a phenomenal networking conundrum. Our GitLab instance was configured internally with an external_url string mapping (e.g., gitlab.internal.com). When the GitLab server assigned the job to the Runner, it commanded it to download the code via that custom DNS string.

However, because the runner spun up an entirely isolated Docker container to execute the job, that specific container lacked the host machine's internal DNS routing tables. The container failed to resolve gitlab.internal.com back to our internal local IP (192.168.1.X) and timed out instantly.

The Architect's Fix (DNS Injection)

Rather than writing messy /etc/hosts entries onto the server OS, I opened the runner's master configuration map (/etc/gitlab-runner/config.toml).

We can solve this natively at the runner orchestration level in one of two ways:

Method A (The Force Route): Forcing the clone_url globally in the configuration ensures the runner skips DNS parsing and hard-routes traffic explicitly to the provided local IP block.

TOML

[[runners]]
  name = "docker-runner-deploy-machine"
  url = "http://192.168.1.X"
  clone_url = "http://192.168.1.X"   # Forced Route Injection

Method B (The Elegant DNS Hack): Instructing the GitLab Runner controller to inject specific extra_hosts data inside the Docker execution pod exactly at the time of creation.

TOML

[runners.docker]
    image = "docker:24.0.5"
    privileged = true
    extra_hosts =["gitlab.internal.com:192.168.1.X"]   # DNS Mapping Injection

Applying this fix to config.toml and executing sudo gitlab-runner restart bridged the networking gap perfectly.

Conclusion

Successfully engineering a CI/CD environment requires significantly more logic than basic CLI utilization; it requires an inherent understanding of network abstraction and daemon isolation. Whether bypassing containerized DNS boundaries through configuration injection or handling container privileges, the value of a Senior Platform Engineer is in recognizing the specific communication roadblocks across a decoupled infrastructure stack.