CI/CD Guide¶

This guide covers continuous integration and deployment for the SPOT Platform.

Overview¶

SPOT uses GitLab CI/CD with a modular pipeline structure. You can test the pipeline locally before pushing to GitLab.

Pipeline Structure¶

The CI/CD pipeline has these stages:

prepare - Build base images (base, devtools)
validate - Validate lockfiles and dependencies
build - Build service Docker images
code-quality - Lint, format check, type check
test - Unit and integration tests
scan - Security scanning
deploy - Deploy to environments (manual)
documentation - Generate docs

Local CI Testing¶

Test the CI pipeline locally using Docker-in-Docker for complete isolation.

Prerequisites¶

Install gitlab-ci-local (required for local CI testing):

npm install -g gitlab-ci-local

For more information: https://github.com/firecow/gitlab-ci-local

Quick Start¶

# Run full CI pipeline
make ci:local

# Run specific job
make ci:local JOB="test-services: [api-gateway]"

# Clean up CI environment
make dev:clean

How It Works¶

Local CI uses Docker-in-Docker (DinD) for isolation:

Host Machine
├── Docker Daemon (dev environment)
│   └── Your development containers      ← Your dev postgres, redis, etc.
│
└── spot-ci-dind Container (privileged)  ← Special container with Docker inside
    └── Docker Daemon (isolated CI)      ← Separate Docker daemon
        └── CI test containers           ← CI postgres, redis, etc.

Benefits:

Complete isolation from development environment
Matches GitLab CI behavior exactly
No interference with dev containers
Automatic cleanup

Local CI Registry¶

The CI pipeline uses a Docker registry (localhost:5000) as an image cache between CI jobs. This registry is only needed for CI testing, not for regular local development.

Purpose: - Cache built images between CI jobs running in Docker-in-Docker - Allow CI jobs to push/pull images just like the real GitLab CI - Test the complete build → push → pull workflow locally

Automatic startup: The make ci:local command automatically starts the registry if not already running. You don't need to start it manually.

Manual control:

# Start registry manually
docker compose --profile ci up -d registry

# Check registry status
curl http://localhost:5000/v2/_catalog

# Stop registry
docker compose --profile ci stop registry

For local development: You don't need the registry. The base image is built automatically when building services:

# Build all services (auto-builds base image if missing)
make docker:build

# Or build base image explicitly
make docker:build-base

Configuration¶

Copy and configure the environment file:

cp .gitlab-ci-local-env.example.dind .gitlab-ci-local-env.dind

Edit .gitlab-ci-local-env.dind:

# GitLab credentials
GITLAB_TOKEN=your-gitlab-token

# Registry (for local testing)
CI_REGISTRY=localhost:5000
CI_REGISTRY_IMAGE=localhost:5000/spot/platform

# Test environment
APP_ENV=test
POSTGRES_DB=spot_ci_test
SECRET_KEY=test-secret-key

Running CI Jobs¶

Run specific jobs:

# Validation jobs
make ci:local JOB=validate-dependencies

# Build jobs
make ci:local JOB=build-base
make ci:local JOB="build-services: [api-gateway]"

# Test jobs
make ci:local JOB="test-services: [api-gateway]"

# Integration tests
make ci:local JOB=test-integration

Run all jobs:

# Run all jobs, including manual jobs
make ci:local FULL=1

Cleanup¶

# Clean up CI environment
make dev:clean

GitLab CI Pipeline¶

Automatic Triggers¶

Pipeline runs automatically on:

Push to any branch
Merge requests
Tags

Manual Jobs¶

Some jobs require manual approval:

deploy-production
security-scan (optional)

Pipeline Configuration¶

Pipeline is defined in modular files:

.gitlab/ci/
├── templates/
│   ├── global.yml      # Global config
│   ├── docker.yml      # Base for jobs using Docker
│   ├── python.yml      # Base for jobs using Python
│   └── security.yml    # Base for security jobs
├── stages/
│   ├── 01-prepare.yml     # Build base images
│   ├── 02-validate.yml    # Validate lockfiles
│   ├── 03-build.yml       # Build service images
│   ├── 04-code-quality.yml # Lint, format, type-check
│   ├── 05-test.yml        # Unit and integration tests
│   ├── 06-scan.yml        # Security scanning
│   ├── 07-deploy.yml      # Deploy to environments
│   └── 08-documentation.yml # Generate docs
└── .gitlab-ci.yml         # Main config (includes all)

Environment Variables¶

Configure in GitLab UI (Settings > CI/CD > Variables):

# Required
GITLAB_TOKEN          # GitLab API token
CI_REGISTRY_PASSWORD  # Registry password

# Optional
CI_REGISTRY           # Registry URL (default: gitlab registry)
SECRET_KEY            # Production secret key
POSTGRES_PASSWORD     # Production database password

Job Examples¶

Validation job:

validate:formatting:
  stage: validate
  script:
    - python -m cli quality format --check

Build job:

build:api-gateway:
  stage: build
  script:
    - python -m cli docker build --service api-gateway
    - docker push $CI_REGISTRY_IMAGE/api-gateway:$CI_COMMIT_SHA

Test job:

test:api-gateway:
  stage: test
  script:
    - python -m cli test unit --service api-gateway
  coverage: '/TOTAL.*\s+(\d+%)$/'

CLI Commands for CI¶

CI jobs use CLI commands for consistency:

# Validation
make quality:format-check           # Check code formatting
make quality:lint                     # Run linting
make quality:type-check               # Run type checking
make ci:validate-dependencies         # Validate poetry.lock files

# Building
make docker:build-base                # Build base image
make docker:build SERVICE=api-gateway   # Build service
make docker:build                     # Build all services

# Testing
make test:unit SERVICE=api-gateway  # Unit tests
make test:integration                 # Integration tests
make test                         # All tests
make test:coverage             # Coverage report

# Documentation
make docs:build                           # Build HTML documentation
make docs:serve                           # Serve docs locally on :8082

Deployment¶

Both staging and production deployments require manual approval.

Staging Deployment¶

Manual deployment to staging (triggered on main or release branches):

deploy-staging:
  stage: deploy
  when: manual
  rules:
    - if: $CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH =~ /^release\//

Production Deployment¶

Manual deployment to production (triggered on semver tags):

deploy-production:
  stage: deploy
  when: manual
  rules:
    - if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/

Best Practices¶

Before Committing¶

Run these checks locally:

# Test the full CI pipeline locally
make ci:local FULL=1

Commit Guidelines¶

Use conventional commits:

feat: Add email attachment analysis
fix: Resolve database connection timeout
docs: Update API documentation
test: Add integration tests for workflows

Merge Request Process¶

Create feature branch
Make changes with tests
Run local checks: make quality:lint && make test
Test CI locally: make ci:local FULL=1
Push and create merge request
Wait for CI pipeline to pass
Request review
Merge after approval

Troubleshooting¶

Local CI Issues¶

CI fails with permission errors:

# Ensure DinD container is running
docker ps | grep spot-ci-dind

# Clean and restart
make dev:clean
make ci:local

CI uses wrong environment:

# Check environment file
cat .gitlab-ci-local-env.dind | grep APP_ENV

# Should be: APP_ENV=test

CI containers conflict with dev:

# Check namespacing
docker ps | grep spot-ci

# All CI containers should have spot-ci prefix
# Dev containers should NOT have this prefix

GitLab CI Issues¶

Pipeline fails on GitLab but passes locally:

Check GitLab environment variables
Verify registry credentials
Review job logs in GitLab UI

Build fails with registry auth error:

Verify CI_REGISTRY_PASSWORD is set
Check GITLAB_TOKEN has registry access
Ensure registry URL is correct

Tests fail in CI but pass locally:

Check environment variables in GitLab
Verify test database is accessible
Review test logs for specific errors

CI Performance¶

Optimization Tips¶

Use Docker layer caching
Run jobs in parallel
Skip unnecessary jobs (use only and except)
Use artifacts to share build outputs
Cache dependencies (Poetry virtualenvs)

Advanced Usage¶

Custom CI Jobs¶

Add custom jobs in .gitlab-ci.yml:

custom:job:
  stage: test
  script:
    - make quality:lint
  only:
    - branches

Manual Triggers¶

Run jobs manually in GitLab UI: 1. Go to CI/CD > Pipelines 2. Click pipeline 3. Find manual job 4. Click play button

Debugging CI¶

Enable debug mode:

variables:
  CI_DEBUG_TRACE: "true"

View job logs: 1. Go to CI/CD > Pipelines 2. Click pipeline 3. Click job 4. View output