Setting Up CI/CD with GitHub Actions Self-Hosted Runner (And Everything That Broke Along the Way)

I finally did it. After weeks of manually SSHing into my production server, pulling Docker images, and restarting containers like it’s 2015, I set up a proper CI/CD pipeline for my hobby project.

The goal was simple: push to main, and everything deploys automatically.

The journey? Not so simple. Here’s everything I learned.

The Setup

My project Sudden is a monorepo with:

Backend: Spring Boot application
Frontend: React application
Infrastructure: MongoDB, Redis, all orchestrated with Docker Compose
Deployment: Single production server running everything

Before this, my deployment process looked like:

git push → SSH into server → docker compose pull → docker compose up -d → pray

Now it’s just:

git push → done

Part 1: Understanding GitHub Actions

Before writing any YAML, I needed to understand what I was working with.

The Core Concepts

Workflows are automated processes you define in YAML files. They live in .github/workflows/ and describe what should happen and when.

Runners are the machines that actually execute your workflows. GitHub provides hosted runners (their machines), but you can also use self-hosted runners (your machines).

Jobs are groups of steps that run on the same runner. Steps are the individual commands or actions within a job.

Why Self-Hosted?

GitHub’s hosted runners are convenient, but I went self-hosted for a few reasons:

Learning opportunity — I wanted to understand the infrastructure
Cost efficiency — My production server was already sitting there
Persistence — Docker images and build caches stick around between runs
Control — Full access to local resources and custom configurations

Part 2: Setting Up the Runner

First Discovery: Private Repos Only

Self-hosted runners require a private repository. This is a security measure — with public repos, anyone could fork your repo and run arbitrary code on your machine. Fair enough.

Creating a Dedicated User

GitHub’s documentation recommends against running the runner as root. So I created a dedicated user:

# Create the user
sudo useradd -m -s /bin/bash github-runner

# Set a password
sudo passwd github-runner

# Add to sudoers if needed for deployment tasks
sudo usermod -aG sudo github-runner

Installing the Runner

From your repository, go to Settings → Actions → Runners → New self-hosted runner. GitHub provides the exact commands to run:

# Switch to the runner user
su - github-runner

# Create and enter the directory
mkdir actions-runner && cd actions-runner

# Download the runner package
curl -o actions-runner-linux-x64-2.331.0.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.331.0/actions-runner-linux-x64-2.331.0.tar.gz

# Extract
tar xzf ./actions-runner-linux-x64-2.331.0.tar.gz

# Configure (GitHub provides a unique token)
./config.sh --url https://github.com/YOUR_USERNAME/YOUR_REPO --token YOUR_TOKEN

# Test it interactively
./run.sh

At this point, the runner shows as “Idle” in GitHub’s UI. But we’re not done — running interactively won’t survive a reboot.

Part 3: The Workflow File

Here’s the complete workflow I ended up with (.github/workflows/deploy.yml):

name: Build and Deploy

on:
  push:
    branches:
      - main

env:
  DOCKERHUB_USERNAME: dkadev
  BACKEND_IMAGE: dkadev/sudden-backend
  FRONTEND_IMAGE: dkadev/sudden-frontend

jobs:
  build-and-deploy:
    runs-on: self-hosted

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ env.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push backend image
        uses: docker/build-push-action@v6
        with:
          context: ./sudden
          file: ./sudden/Dockerfile
          push: true
          tags: |
            ${{ env.BACKEND_IMAGE }}:latest
            ${{ env.BACKEND_IMAGE }}:${{ github.sha }}

      - name: Build and push frontend image
        uses: docker/build-push-action@v6
        with:
          context: ./sudden-client/trading-journal
          file: ./sudden-client/trading-journal/Dockerfile
          push: true
          tags: |
            ${{ env.FRONTEND_IMAGE }}:latest
            ${{ env.FRONTEND_IMAGE }}:${{ github.sha }}

      - name: Deploy with Docker Compose
        working-directory: ${{ github.workspace }}
        env:
          # Secrets (sensitive)
          MONGO_INITDB_ROOT_PASSWORD: ${{ secrets.MONGO_INITDB_ROOT_PASSWORD }}
          MONGO_APP_PASSWORD: ${{ secrets.MONGO_APP_PASSWORD }}
          # Variables (non-sensitive)
          MONGO_INITDB_ROOT_USERNAME: ${{ vars.MONGO_INITDB_ROOT_USERNAME }}
          MONGO_DATABASE: ${{ vars.MONGO_DATABASE }}
          MONGO_APP_USERNAME: ${{ vars.MONGO_APP_USERNAME }}
          REDIS_HOST: ${{ vars.REDIS_HOST }}
          REDIS_PORT: ${{ vars.REDIS_PORT }}
          SERVER_PORT: ${{ vars.SERVER_PORT }}
          FRONTEND_PORT: ${{ vars.FRONTEND_PORT }}
        run: |
          docker compose pull backend frontend
          docker compose up -d backend frontend

Looks clean, right? Getting here was anything but.

Part 4: The Troubleshooting Gauntlet

This is where the real learning happened. Seven issues, seven solutions.

Issue 1: Docker Permission Denied

The error:

permission denied while trying to connect to the Docker daemon socket at
unix:///var/run/docker.sock

What happened: The github-runner user didn’t have permission to talk to Docker.

The fix:

sudo usermod -aG docker github-runner

The catch: Group changes don’t apply to running processes. You need to log out and back in, or restart the service. This bit me later.

Issue 2: Runner Needs to Run as a Service

Running ./run.sh interactively is fine for testing, but it dies when you close the terminal and doesn’t survive reboots.

The fix: Use the built-in service installer:

# Install as a systemd service
sudo ./svc.sh install github-runner

# Start it
sudo ./svc.sh start

# Check status
sudo ./svc.sh status

Now it starts automatically on boot. Other useful commands:

sudo ./svc.sh stop      # Stop the service
sudo ./svc.sh start     # Start the service
sudo ./svc.sh status    # Check status

# View logs
journalctl -u actions.runner.YOUR-REPO.YOUR-MACHINE.service -f

Issue 3: Docker Compose Not Found

The error:

docker: unknown command: docker compose

What happened: The server had the old standalone docker-compose (v1) but not docker compose (v2, the Docker CLI plugin).

The fix: Install Docker Compose v2 as a CLI plugin:

# Create plugin directory
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins

# Download the plugin
curl -SL https://github.com/docker/compose/releases/download/v2.32.2/docker-compose-linux-x86_64 \
  -o $DOCKER_CONFIG/cli-plugins/docker-compose

# Make it executable
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

# Install system-wide (for all users including github-runner)
sudo mkdir -p /usr/local/lib/docker/cli-plugins
sudo cp $DOCKER_CONFIG/cli-plugins/docker-compose /usr/local/lib/docker/cli-plugins/

Issue 4: Docker Buildx Not Found

The error:

docker: unknown command: docker buildx

What happened: The docker/setup-buildx-action in the workflow expects the Buildx plugin to be available.

The fix:

# Get the latest version
BUILDX_VERSION=$(curl -s https://api.github.com/repos/docker/buildx/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")')

# Download it
curl -SL "https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64" \
  -o /tmp/docker-buildx

# Install system-wide
sudo mv /tmp/docker-buildx /usr/local/lib/docker/cli-plugins/docker-buildx
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx

Issue 5: The Mysterious “invalid proto:” Error

The error:

time="2026-01-17T14:16:19Z" level=warning msg="...version is obsolete..."
invalid proto:
Error: Process completed with exit code 1.

This one was frustrating. No clear indication of what “proto” meant or why it was invalid.

The diagnosis: I used strace to trace what was happening:

strace -f docker compose config 2>&1 | grep -i proto

This revealed the error was coming from OpenTelemetry internals — a bug in Docker Compose v5.0.1.

The fix: Downgrade to a stable version:

# Download the stable version
curl -SL "https://github.com/docker/compose/releases/download/v2.32.2/docker-compose-linux-x86_64" \
  -o /tmp/docker-compose-v2

# Replace everywhere
chmod +x /tmp/docker-compose-v2
sudo cp /tmp/docker-compose-v2 ~/.docker/cli-plugins/docker-compose
sudo cp /tmp/docker-compose-v2 /usr/local/lib/docker/cli-plugins/docker-compose
sudo cp /tmp/docker-compose-v2 /home/github-runner/.docker/cli-plugins/docker-compose
sudo chown github-runner:github-runner /home/github-runner/.docker/cli-plugins/docker-compose

# Verify
docker compose version  # Should show v2.32.2

Sometimes newer isn’t better.

Issue 6: Environment Variables Not Set

The error:

The "FRONTEND_PORT" variable is not set. Defaulting to a blank string.
no port specified: 8082:<empty>

What happened: I had added all the secrets and variables to GitHub’s settings, but the workflow wasn’t receiving them.

The key learning: Adding secrets to GitHub isn’t enough. You must explicitly pass them to each step that needs them.

The fix: Add an env: block to the deployment step:

- name: Deploy with Docker Compose
  env:
    MONGO_INITDB_ROOT_PASSWORD: ${{ secrets.MONGO_INITDB_ROOT_PASSWORD }}
    SERVER_PORT: ${{ vars.SERVER_PORT }}
    # ... other variables
  run: |
    docker compose up -d

Important distinction:

${{ secrets.NAME }} — For sensitive data (passwords, tokens)
${{ vars.NAME }} — For non-sensitive configuration (ports, hostnames)

Issue 7: Stale Runner Process

The symptom: After adding the user to the docker group and restarting the service, permission errors still persisted.

What happened: An old runner process from before the group change was still running in the background.

The fix: Kill all old processes and restart fresh:

# Find old processes
ps aux | grep Runner.Listener

# Kill them
sudo pkill -f Runner.Listener

# Restart the service
sudo ./svc.sh start

# Verify the new process has correct groups
ps -o pid,user,group,supgrp,cmd -p $(pgrep -f Runner.Listener)
# Should show: docker,github-runner in SUPGRP column

Part 5: GitHub Secrets and Variables

A quick reference on setting these up.

Secrets (Sensitive Data)

Go to Repository → Settings → Secrets and variables → Actions → New repository secret.

My secrets:

DOCKERHUB_TOKEN — Docker Hub access token
MONGO_INITDB_ROOT_PASSWORD — MongoDB root password
MONGO_APP_PASSWORD — Application database password

Variables (Non-Sensitive Configuration)

Same location, but under the Variables tab.

My variables:

MONGO_INITDB_ROOT_USERNAME
MONGO_DATABASE
MONGO_APP_USERNAME
REDIS_HOST
REDIS_PORT
SERVER_PORT
FRONTEND_PORT

Part 6: Debugging Failed Workflows

When things go wrong (and they will), here’s where to look.

Log Locations

1. Systemd journal (service-level logs):

journalctl -u actions.runner.YOUR-REPO.YOUR-MACHINE.service -f

2. Runner diagnostic logs:

ls -lt /home/github-runner/actions-runner/_diag/

Two types of files here:

Runner_*.log — Main runner process logs
Worker_*.log — Individual job execution logs

3. View the latest job log:

tail -f /home/github-runner/actions-runner/_diag/Worker_20260117-141353-utc.log

The Final Docker Compose File

For reference, here’s my docker-compose.yml:

services:
  mongodb:
    image: mongo:7.0
    container_name: sudden-mongodb
    restart: unless-stopped
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
      MONGO_INITDB_DATABASE: ${MONGO_DATABASE}
    ports:
      - "27018:27017"
    volumes:
      - mongodb_data:/data/db
    networks:
      - sudden-network

  redis:
    image: redis:7-alpine
    container_name: sudden-redis
    restart: unless-stopped
    ports:
      - "6380:6379"
    volumes:
      - redis_data:/data
    networks:
      - sudden-network

  backend:
    image: dkadev/sudden-backend:latest
    container_name: sudden-backend
    restart: unless-stopped
    environment:
      SPRING_DATA_MONGODB_URI: mongodb://${MONGO_APP_USERNAME}:${MONGO_APP_PASSWORD}@mongodb:27017/${MONGO_DATABASE}
      SPRING_REDIS_HOST: ${REDIS_HOST}
      SPRING_REDIS_PORT: ${REDIS_PORT}
      SERVER_PORT: ${SERVER_PORT}
    ports:
      - "8082:${SERVER_PORT}"
    depends_on:
      mongodb:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - sudden-network

  frontend:
    image: dkadev/sudden-frontend:latest
    container_name: sudden-frontend
    restart: unless-stopped
    ports:
      - "${FRONTEND_PORT}:80"
    depends_on:
      - backend
    networks:
      - sudden-network

networks:
  sudden-network:
    driver: bridge

volumes:
  mongodb_data:
  redis_data:

Key Takeaways

Nine things I’ll remember:

Self-hosted runners need private repos — Security requirement from GitHub
Never run as root — Create a dedicated user for the runner
Service mode is essential — Use svc.sh to run the runner as a systemd service
Docker permissions matter — Add the runner user to the docker group
Plugin versions can break things — Sometimes downgrading is the fix (docker-compose v5.0.1 bug)
Secrets vs Variables — Use the correct syntax (secrets.* vs vars.*)
Environment variables must be passed explicitly — Adding to GitHub isn’t enough
Restart services after group changes — Group memberships are read at process start
Know your log locations — Essential for debugging failed workflows

Wrapping Up

What started as “I’ll just automate my deployments” turned into a deep dive into Linux user management, Docker internals, systemd services, and production debugging.

Was it worth the effort? Every push to main now builds, pushes, and deploys without me lifting a finger. So yes, absolutely.

The Setup#

Part 1: Understanding GitHub Actions#

The Core Concepts#

Why Self-Hosted?#

Part 2: Setting Up the Runner#

First Discovery: Private Repos Only#

Creating a Dedicated User#

Installing the Runner#

Part 3: The Workflow File#

Part 4: The Troubleshooting Gauntlet#

Issue 1: Docker Permission Denied#

Issue 2: Runner Needs to Run as a Service#

Issue 3: Docker Compose Not Found#

Issue 4: Docker Buildx Not Found#

Issue 5: The Mysterious “invalid proto:” Error#

Issue 6: Environment Variables Not Set#

Issue 7: Stale Runner Process#

Part 5: GitHub Secrets and Variables#

Secrets (Sensitive Data)#

Variables (Non-Sensitive Configuration)#

Part 6: Debugging Failed Workflows#

Log Locations#

The Final Docker Compose File#

Key Takeaways#

Wrapping Up#

The Setup

Part 1: Understanding GitHub Actions

The Core Concepts

Why Self-Hosted?

Part 2: Setting Up the Runner

First Discovery: Private Repos Only

Creating a Dedicated User

Installing the Runner

Part 3: The Workflow File

Part 4: The Troubleshooting Gauntlet

Issue 1: Docker Permission Denied

Issue 2: Runner Needs to Run as a Service

Issue 3: Docker Compose Not Found

Issue 4: Docker Buildx Not Found

Issue 5: The Mysterious “invalid proto:” Error

Issue 6: Environment Variables Not Set

Issue 7: Stale Runner Process

Part 5: GitHub Secrets and Variables

Secrets (Sensitive Data)

Variables (Non-Sensitive Configuration)

Part 6: Debugging Failed Workflows

Log Locations

The Final Docker Compose File

Key Takeaways

Wrapping Up