CI/CD Guide¶

This guide covers continuous integration and deployment for the SPOT Platform.

Overview¶

SPOT uses GitLab CI/CD with a modular pipeline structure. You can test the pipeline locally before pushing to GitLab.

Pipeline Structure¶

The CI/CD pipeline has these stages:

1. prepare - Build base images (base, devtools)
2. validate - Validate lockfiles and dependencies
3. build - Build service Docker images
4. code-quality - Lint, format check, type check
5. test - Unit and integration tests
6. scan - Security scanning
7. deploy - Publish a tag's images to Docker Hub, then check them
8. release - Mirror the tag to Codeberg and deploy to staging (manual), once the release is published and checked
9. documentation - Build the documentation site, and publish a release's site to Codeberg Pages

One Pipeline for Three Parts¶

The repository holds core, the web dashboard (web-dashboard/) and the deployment files (deploy/). One .gitlab-ci.yml runs them all, and each part's jobs run only when that part changed:

A tag runs every job. secret-scan (core's code, informative: it never fails), parts-secret-scan (the web dashboard, deploy/ and docs/, blocking), audit (a new advisory needs no change to appear), resolve-devtools-digest (it names the devtools image that secret-scan and openapi run in) and plumber's job run on every pipeline. A change that touches one part therefore shows that part's jobs and those repository-wide ones only. parts-secret-scan scans the files, not the history; the history holds the deploy smoke test's throwaway key, which .gitleaksignore allows for a scan of the history.

The rules live in .rules-core (.gitlab/ci/templates/global.yml), .rules-dashboard and .rules-deploy-files.

The jobs of the web dashboard:

• dashboard-python: spot/ci-templates' python component: the lockfile, the runtime dependencies' licences, format, lint, types and tests with the 100 % coverage gate;
• dashboard-frontend-check, dashboard-validate-dockerfile;
• dashboard-build-image: spot/ci-templates' image component, from the repository root (the image installs the dashboard from uv.lock), after core's builds;
• dashboard-scan-container (Trivy, default branch and tags, with web-dashboard/.trivyignore), dashboard-scan-sast and dashboard-trivyignore-reminder.

The repository's audit reads uv.lock, which holds the dashboard's packages too.

The jobs of the deployment files:

• deploy-compose, deploy-env, deploy-scripts and deploy-core-sync check the compose file, its environment documentation, the scripts and the compose file's agreement with core's compose/;
• deploy-external-images (a tag, or a merge request that changes the compose file) checks that every image the compose file names and SPOT does not publish exists, before publish copies anything;
• deploy-images checks the tag's own images on Docker Hub once publish has run (manual on main), and deploy-smoke boots the stack (automatic on a tag, manual on main and on a merge request that changes the stack);

The documentation site¶

The three parts' documentation is one MkDocs site (mkdocs.yml, the pages in docs/: the platform's at the root, docs/deployment/ and docs/web-ui/; the SDK's pages are imported from its repository at build time). pages builds it strictly (scripts/docs-build.sh, which make docs:build runs too, so a local pass is a CI pass) on a merge request and on the default branch when docs/, mkdocs.yml, the build script or the docs group of the lock changed, and on every tag. The build fails if the job token's value appears anywhere in public/.

The public site documents the latest release, not main, which is development: publish-codeberg-pages runs on tags only, after deploy-smoke and mirror-codeberg have passed, and publishes public/ to the pages branch of SPOT_Project/documentation on Codeberg, which serves it at https://spot-project.codeberg.page/documentation/. A documentation fix therefore reaches the public site with the next tag. The Quick Start checks out the latest release tag for the same reason. The old /platform/ URLs redirect (mkdocs-redirects).

The SDK's pages are cloned from its repository when the site is built, so a change to the SDK's docs reaches the site with the next tag (or the next change of the files above). The dependencies are the docs group of pyproject.toml; serve the site locally with make docs:serve.

Where the images come from¶

On a tag, the builds push ${CI_REGISTRY_IMAGE}/<image>:<version>. One publish job (the publish component of spot/ci-templates) then copies the seven images to docker.io/spotproject/<image>:<version>, and moves :latest for a release but not for a pre-release. It has no needs:, so it waits for every earlier stage: nothing reaches Docker Hub unless every check passed. deploy-images and deploy-smoke then check what it published. mirror-codeberg (the mirror component, in the release stage) pushes main and the tags to Codeberg, a tag only after those checks have passed: Codeberg's tags are what the update banner and spot-cli upgrade read.

Image Tags¶

Images built by CI are pushed to the project registry by scripts/ci-image.sh so concurrent pipelines never share a tag:

Jobs pull the image built for their own commit (ci_image_pull). When the pipeline did not rebuild an image (nothing it depends on changed), the job falls back to :latest and logs which reference it used. SPOT_BASE_DIGEST and SPOT_DEVTOOLS_DIGEST always point at the image the pipeline pushed, or at :latest through resolve-*-digest when it was not rebuilt.

Local CI Testing¶

Test the CI pipeline locally using Docker-in-Docker for complete isolation.

Prerequisites¶

Install gitlab-ci-local (required for local CI testing):

npm install -g gitlab-ci-local

For more information: https://github.com/firecow/gitlab-ci-local

Quick Start¶

# Run full CI pipeline
make ci:local

# Run specific job
make ci:local JOB="test-services: [api-gateway]"

# Clean up CI environment
make dev:clean

How It Works¶

Local CI uses Docker-in-Docker (DinD) for isolation:

Host Machine
├── Docker Daemon (dev environment)
│   └── Your development containers      ← Your dev postgres, redis, etc.
│
└── spot-ci-dind Container (privileged)  ← Special container with Docker inside
    └── Docker Daemon (isolated CI)      ← Separate Docker daemon
        └── CI test containers           ← CI postgres, redis, etc.

Benefits:

• Complete isolation from development environment
• Matches GitLab CI behavior exactly
• No interference with dev containers
• Automatic cleanup

Local CI Registry¶

The CI pipeline uses a Docker registry (localhost:5000) as an image cache between CI jobs. This registry is only needed for CI testing, not for regular local development.

Purpose: - Cache built images between CI jobs running in Docker-in-Docker - Allow CI jobs to push/pull images just like the real GitLab CI - Test the complete build → push → pull workflow locally

Automatic startup: The make ci:local command automatically starts the registry if not already running. You don't need to start it manually.

Manual control:

# Start registry manually
docker compose --profile ci up -d registry

# Check registry status
curl http://localhost:5000/v2/_catalog

# Stop registry
docker compose --profile ci stop registry

For local development: You don't need the registry. The base image is built automatically when building services:

# Build all services (auto-builds base image if missing)
make docker:build

# Or build base image explicitly
make docker:build-base

Configuration¶

Copy and configure the environment file:

cp .gitlab-ci-local-env.example.dind .gitlab-ci-local-env.dind

Edit .gitlab-ci-local-env.dind:

# GitLab credentials
GITLAB_TOKEN=your-gitlab-token

# Registry (for local testing)
CI_REGISTRY=localhost:5000
CI_REGISTRY_IMAGE=localhost:5000/spot/platform

# Test environment
APP_ENV=test
POSTGRES_DB=spot_ci_test
SECRET_KEY=test-secret-key

Running CI Jobs¶

Run specific jobs:

# Validation jobs
make ci:local JOB=audit

# Build jobs
make ci:local JOB=build-base
make ci:local JOB="build-services: [api-gateway]"

# Test jobs
make ci:local JOB="test-services: [api-gateway]"

# Integration tests
make ci:local JOB=test-integration

Run all jobs:

# Run all jobs, including manual jobs
make ci:local FULL=1

Cleanup¶

# Clean up CI environment
make dev:clean

GitLab CI Pipeline¶

Automatic Triggers¶

Pipeline runs automatically on:

• Push to any branch
• Merge requests
• Tags

Manual Jobs¶

Some jobs require manual approval:

• deploy-production
• security-scan (optional)

Pipeline Configuration¶

Pipeline is defined in modular files:

.gitlab/ci/
├── templates/
│   ├── global.yml      # Global config
│   ├── docker.yml      # Base for jobs using Docker
│   ├── python.yml      # Base for jobs using Python
│   └── security.yml    # Base for security jobs
├── stages/
│   ├── 01-prepare.yml     # Build base images
│   ├── 02-validate.yml    # Validate lockfiles
│   ├── 03-build.yml       # Build service images
│   ├── 04-code-quality.yml # Lint, format, type-check
│   ├── 05-test.yml        # Unit and integration tests
│   ├── 06-scan.yml        # Security scanning
│   ├── 07-deploy.yml      # Deploy to environments
│   └── 08-documentation.yml # Generate docs
└── .gitlab-ci.yml         # Main config (includes all)

Environment Variables¶

Configure in GitLab UI (Settings > CI/CD > Variables):

# Required
GITLAB_TOKEN          # GitLab API token
CI_REGISTRY_PASSWORD  # Registry password

# Optional
CI_REGISTRY           # Registry URL (default: gitlab registry)
SECRET_KEY            # Production secret key
POSTGRES_PASSWORD     # Production database password

Job Examples¶

Validation job:

validate:formatting:
  stage: validate
  script:
    - python -m cli quality format --check

Build job:

build:api-gateway:
  stage: build
  script:
    - python -m cli docker build --service api-gateway
    - docker push $CI_REGISTRY_IMAGE/api-gateway:$CI_COMMIT_SHA

Test job:

test:api-gateway:
  stage: test
  script:
    - python -m cli test unit --service api-gateway
  coverage: '/TOTAL.*\s+(\d+%)$/'

CLI Commands for CI¶

CI jobs use CLI commands for consistency:

# Validation
make quality:format-check           # Check code formatting
make quality:lint                     # Run linting
make quality:type-check               # Run type checking
make ci:validate-dependencies         # Audit uv.lock locally, as the CI's audit does

# Building
make docker:build-base                # Build base image
make docker:build SERVICE=api-gateway   # Build service
make docker:build                     # Build all services

# Testing
make test:unit SERVICE=api-gateway  # Unit tests
make test:integration                 # Integration tests
make test                         # All tests
make test:coverage             # Coverage report

# Documentation
make docs:build                           # Build the site into public/, strictly, as CI does
make docs:serve                           # Serve docs locally on :8082
make docs:openapi                         # Generate docs/api from the code (OUTPUT=dir writes elsewhere)

Deployment¶

Both staging and production deployments require manual approval.

Staging Deployment¶

Manual deployment to staging, on a version tag. The job runs spot-cli upgrade <version> in /opt/spot/deploy on the staging host, which backs up first and waits for every health check:

deploy-staging:
  stage: release
  needs: [publish, mirror-codeberg]
  rules:
    - if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$/
      when: manual
      allow_failure: true

Production Deployment¶

Manual deployment to production (triggered on semver tags):

deploy-production:
  stage: deploy
  when: manual
  rules:
    - if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/

Best Practices¶

Before Committing¶

Run these checks locally:

# Test the full CI pipeline locally
make ci:local FULL=1

Commit Guidelines¶

Use conventional commits:

feat: Add email attachment analysis
fix: Resolve database connection timeout
docs: Update API documentation
test: Add integration tests for workflows

Merge Request Process¶

1. Create feature branch
2. Make changes with tests
3. Run local checks: make quality:lint && make test
4. Test CI locally: make ci:local FULL=1
5. Push and create merge request
6. Wait for CI pipeline to pass
7. Request review
8. Merge after approval

Troubleshooting¶

Local CI Issues¶

CI fails with permission errors:

# Ensure DinD container is running
docker ps | grep spot-ci-dind

# Clean and restart
make dev:clean
make ci:local

CI uses wrong environment:

# Check environment file
cat .gitlab-ci-local-env.dind | grep APP_ENV

# Should be: APP_ENV=test

CI containers conflict with dev:

# Check namespacing
docker ps | grep spot-ci

# All CI containers should have spot-ci prefix
# Dev containers should NOT have this prefix

GitLab CI Issues¶

Pipeline fails on GitLab but passes locally:

• Check GitLab environment variables
• Verify registry credentials
• Review job logs in GitLab UI

Build fails with registry auth error:

• Verify CI_REGISTRY_PASSWORD is set
• Check GITLAB_TOKEN has registry access
• Ensure registry URL is correct

Tests fail in CI but pass locally:

• Check environment variables in GitLab
• Verify test database is accessible
• Review test logs for specific errors

CI Performance¶

Optimization Tips¶

1. Use Docker layer caching
2. Run jobs in parallel
3. Skip unnecessary jobs (use only and except)
4. Use artifacts to share build outputs
5. Cache dependencies

Advanced Usage¶

Custom CI Jobs¶

Add custom jobs in .gitlab-ci.yml:

custom:job:
  stage: test
  script:
    - make quality:lint
  only:
    - branches

Manual Triggers¶

Run jobs manually in GitLab UI: 1. Go to CI/CD > Pipelines 2. Click pipeline 3. Find manual job 4. Click play button

Debugging CI¶

Enable debug mode:

variables:
  CI_DEBUG_TRACE: "true"

View job logs: 1. Go to CI/CD > Pipelines 2. Click pipeline 3. Click job 4. View output

Resources¶

• GitLab CI/CD Documentation
• Docker-in-Docker
• gitlab-ci-local
• Configuration Reference