~blogcase-study-deploying-behind-nat

Deploying behind NAT: CI/CD to a private server with no open SSH port

2025-01-14 · 6 min read · case study · ci-cd · tailscale · docker · cloudflare

The target server sits behind NAT with nothing published to the internet. No port 22, no jump host with a public IP, no VPN concentrator to run. The CI runner is a disposable container in someone else’s cloud that has never seen my network. It needs to deploy to that box on every push to staging.

The pipeline joins a private mesh network as a tagged machine identity, deploys over it, and disappears. Public traffic to the deployed application arrives through a separate tunnel that also opens no ports. I have run variants of this on both GitHub Actions and Bitbucket Pipelines.

The problem with the normal answer

The default is to open port 22 to the CI provider’s address range and call it segmented. Those ranges are large, they change, and every other tenant of that CI provider is inside them. You have authenticated a network, not a machine.

The other default is a bastion host. That is a public server whose entire job is to be attacked, plus a second SSH hop to debug when a deploy fails at 11pm.

I wanted the deploy path to be closed by default, with the runner holding an identity rather than an address.

Architecture

  git push staging

        v
  ┌──────────────────────────────────────────┐
  │  CI runner (ephemeral)                   │
  │   1. test + build                        │
  │   2. build images                        │
  │   3. push to registry                    │
  │   4. join mesh network as tag:ci         │
  │   5. copy docker-compose.prod.yml        │
  │   6. ssh + compose pull && up -d         │
  └────────────────────┬─────────────────────┘
                       │  private mesh, no public port
                       v
  ┌──────────────────────────────────────────┐
  │  private Docker host  (behind NAT)       │
  │   holds: compose file + .env  only       │
  └────────────────────┬─────────────────────┘
                       │  outbound tunnel connector
                       v
             public HTTPS to the application

Two separate private paths, deliberately. The mesh network carries administrative access for CI. The tunnel carries public application traffic. Neither one requires an inbound firewall rule, and compromising the public path does not hand anybody a deploy channel.

Decisions worth defending

The production server does not contain the git repository. It holds docker-compose.prod.yml, a .env, and credentials to pull images. That is it. No source, no build toolchain, no npm cache, no git pull in a deploy script. The artifact that runs in production is the artifact CI built and tested, not something reconstructed on the target.

Every image gets two tags.

docker build -f apps/api/Dockerfile \
  -t "$REGISTRY/store:api-staging" \
  -t "$REGISTRY/store:api-$SHORT_SHA" .

The moving tag (api-staging) is what the compose file references, so a normal deploy is pull and up -d. The immutable commit tag is what I pin to when I need to roll back to a specific commit or prove which code is running. One tag alone gives you either convenient deploys or traceability, not both.

CI gets a machine identity, not a user account. The runner joins the mesh tagged tag:ci. The server is tag:server. Policy permits tag:ci to reach tag:server on SSH and nothing else, as one named Unix user. No personal credential is involved anywhere in an unattended deployment, which means offboarding a person never breaks a deploy and a leaked deploy key does not grant a human’s access.

Runner identity is ephemeral. Short-lived auth so a runner that vanishes mid-job does not leave a permanently authorised machine behind.

Porting it between CI platforms

I have moved this pipeline in both directions between GitHub Actions and Bitbucket Pipelines. The lesson from doing it twice: map responsibilities, do not translate YAML line by line.

Responsibility GitHub Actions Bitbucket Pipelines
Commit identity github.sha BITBUCKET_COMMIT
Build number github.run_number BITBUCKET_BUILD_NUMBER
Branch trigger on.push.branches pipelines.branches
Environment scoping Environments Deployments
Reusable step Marketplace action Pipe or plain shell
Passing files between jobs Artifacts Step artifacts

Once the table exists the migration is mechanical. Translating syntax first produces a pipeline that runs and quietly does something different, usually around which secrets are visible to which step.

One migration surfaced a security problem rather than a syntax one. The original pipeline generated a .env and an .npmrc in one step and passed them to the next as artifacts. Those files contain credentials, and an artifact is a stored, downloadable object with a retention period. The correct fix is to generate secret-bearing files inside the job that consumes them, keep retention short where artifacts are unavoidable, and prefer build-time secret mounts over files on disk. A line-by-line port would have carried that straight across.

The six ways it broke

Every one of these cost me time, and none of them is in the happy-path documentation:

Failure What it looks like Why
Runner online, SSH refused policy does not permit you to SSH Network ACL and SSH policy are separate layers
Stale mesh node Deploy hangs or targets nothing Old ephemeral runner never went away
Registry hostname fails lookup ... connection refused Host DNS resolver, not the registry
Compose file drift Deploy succeeds, wrong thing runs Server had an older copy
One image tag missing Partial deploy, mixed versions Build for one service failed quietly
.env out of sync Container starts and misbehaves Runtime config on the server predates the compose change

The pattern across all six: the deployment reported success at the layer it could see. Only the last of them is a code problem.

Result

Deploys run on push to staging with no inbound firewall rule on the target host and no source code on it. Rollback is retagging to a known commit sha. When something fails, it fails in a way I can attribute to a layer, because the layers are separate on purpose.

Lessons

The strongest single idea here is that the deploy path should authenticate a machine, not a network location. Address-based access control degrades the moment addresses are shared, and CI address ranges are shared with everyone.

Two smaller ones that keep paying off. Keeping the production server dumb - compose file, env file, nothing else - removes an entire class of “it works on the server but not from CI” confusion, because there is no second way to build anything. And immutable tags alongside moving tags cost one extra docker push and are the only reason a rollback is a thirty-second operation.

Technologies: Docker, Docker Compose, GitHub Actions, Bitbucket Pipelines, Tailscale, Cloudflare Tunnel, private container registry.