GitLab CI/CD Pipeline Best Practices
Overview
GitLab CI/CD pipelines automate the entire software delivery process, from code commit to production deployment. A well-designed pipeline acts as a quality gatekeeper, ensuring that only tested, secure, and properly validated code reaches production environments. This guide covers essential pipeline patterns, optimization techniques, and quality enforcement strategies.
Effective CI/CD pipelines provide rapid feedback to developers - ideally completing pull request validations in under 10 minutes. This quick turnaround prevents context switching and maintains developer productivity. At the same time, pipelines must be thorough, running comprehensive test suites, security scans, and quality checks that would be impractical to execute manually.
The pipeline structure presented here follows a staged approach where each stage has a specific purpose and builds upon the results of previous stages. Stages run sequentially, but jobs within a stage can execute in parallel, maximizing resource utilization and minimizing total pipeline duration.
Core Principles
- Fast Feedback: PR pipelines complete in < 10 minutes
- Quality Gates: Enforce coverage, mutation score, security scans
- Fail Fast: Stop pipeline on first critical failure
- Parallel Execution: Run independent jobs concurrently
- Immutable Artifacts: Build once, deploy many times
- Zero Downtime Deployments: Blue-green or rolling updates
- Audit Trail: Track all deployments, approvals, rollbacks
Pipeline Structure
A well-structured pipeline separates concerns into distinct stages, each with a specific responsibility. This separation enables faster failure detection (fail fast principle) and clearer understanding of where problems occur.
Understanding Pipeline Stages
Stages represent logical groupings of related activities in your delivery process. GitLab executes stages sequentially - all jobs in the current stage must complete before the next stage begins. This creates natural checkpoints where the pipeline can halt if quality standards aren't met.
The validate stage runs first because format and lint checks are fast (typically completing in seconds) and catch obvious problems early. There's no point compiling code that violates basic style rules or has syntax errors.
The build stage compiles source code and creates initial artifacts. Compilation errors surface here, preventing unnecessary test execution. Building once and reusing artifacts in later stages (the "build once, deploy many" principle) ensures consistency - you deploy exactly what you tested.
The test stage runs various test suites. Unit tests, integration tests, and mutation tests typically run in parallel within this stage since they're independent. Parallelization significantly reduces total pipeline time. For an in-depth exploration of testing strategies, see CI Testing.
The quality stage analyzes test results and code quality metrics. Coverage reports from the test stage are processed here to enforce minimum thresholds. SonarQube performs static analysis to detect code smells, bugs, and security vulnerabilities. These quality gates prevent low-quality code from progressing.
The contract stage validates API contracts between services using tools like Pact (see API Contract Testing). This stage runs after quality checks because contract tests assume the code itself is already validated.
The package stage creates deployment artifacts - typically Docker images tagged with version information. The image built here is immutable and will be deployed to all environments without rebuilding, ensuring consistency.
Recommended Stages
# .gitlab-ci.yml
stages:
- validate # Format check, lint, commit message validation
- build # Compile application, build Docker image
- test # Unit tests, integration tests, mutation tests
- quality # Code coverage, SonarQube, security scans
- contract # Contract testing (provider/consumer)
- package # Create deployment artifacts
- deploy-dev # Deploy to development environment
- deploy-uat # Deploy to UAT environment
- deploy-prod # Deploy to production environment
Pipeline Visualization
Basic Pipeline Configuration
Complete Pipeline Example
# .gitlab-ci.yml
default:
image: maven:3.9-eclipse-temurin-21
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .m2/repository
- node_modules/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
DOCKER_DRIVER: overlay2
stages:
- validate
- build
- test
- quality
- package
- deploy
# Validate stage
format-check:
stage: validate
script:
- ./gradlew spotlessCheck --no-daemon
allow_failure: false
lint:
stage: validate
script:
- ./gradlew checkstyleMain checkstyleTest --no-daemon
allow_failure: false
# Build stage
build:
stage: build
script:
- ./gradlew classes --no-daemon
artifacts:
paths:
- build/classes
expire_in: 1 hour
# Test stage (parallel execution)
unit-tests:
stage: test
script:
- ./gradlew test --no-daemon
coverage: '/Total coverage: (\d+\.\d+)%/'
artifacts:
reports:
junit: build/test-results/test/TEST-*.xml
coverage_report:
coverage_format: cobertura
path: build/reports/cobertura/coverage.xml
paths:
- build/test-results
expire_in: 1 week
integration-tests:
stage: test
services:
- postgres:16
- redis:7
variables:
POSTGRES_DB: testdb
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
script:
- ./gradlew integrationTest --no-daemon
artifacts:
reports:
junit: build/test-results/integrationTest/TEST-*.xml
expire_in: 1 week
mutation-tests:
stage: test
script:
- ./gradlew pitest --no-daemon
artifacts:
paths:
- build/reports/pitest
expire_in: 1 week
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == "main"'
# Quality stage
code-coverage:
stage: quality
script:
- ./gradlew jacocoTestReport --no-daemon
- |
COVERAGE=$(awk -F"," '{ instructions += $4 + $5; covered += $5 } END { print 100*covered/instructions }' build/reports/jacoco/test/jacocoTestReport.csv)
if (( $(echo "$COVERAGE < 85" | bc -l) )); then
echo "Coverage $COVERAGE% is below 85% threshold"
exit 1
fi
coverage: '/Total coverage: (\d+\.\d+)%/'
sonarqube:
stage: quality
script:
- ./gradlew sonar --no-daemon
-Dsonar.projectKey=$CI_PROJECT_NAME
-Dsonar.host.url=$SONAR_HOST_URL
-Dsonar.token=$SONAR_TOKEN
-Dsonar.qualitygate.wait=true
only:
- merge_requests
- main
security-scan:
stage: quality
script:
- ./gradlew dependencyCheckAnalyze --no-daemon
artifacts:
reports:
dependency_scanning: build/reports/dependency-check-report.json
allow_failure: false
# Package stage
build-docker:
stage: package
image: docker:24
services:
- docker:24-dind
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
only:
- main
- tags
# Deploy stages
deploy-dev:
stage: deploy
script:
- kubectl set image deployment/payment-service payment-service=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
environment:
name: development
url: https://dev.example.com
only:
- main
deploy-uat:
stage: deploy
script:
- kubectl set image deployment/payment-service payment-service=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
environment:
name: uat
url: https://uat.example.com
when: manual
only:
- main
deploy-prod:
stage: deploy
script:
- kubectl set image deployment/payment-service payment-service=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
environment:
name: production
url: https://api.example.com
when: manual
only:
- tags
Parallel Execution
Parallel execution dramatically reduces pipeline duration by running independent jobs simultaneously. GitLab's parallel keyword and job dependency system (needs) provide fine-grained control over execution order.
Understanding Parallelization
By default, all jobs in a stage run in parallel. However, you can further parallelize individual jobs using the parallel matrix feature, which creates multiple job instances with different variable values. This is particularly useful for splitting large test suites across multiple runners.
The needs keyword allows jobs to start before all jobs in previous stages complete, creating a Directed Acyclic Graph (DAG) of job dependencies. This breaks the strict stage sequencing and enables earlier starts for jobs whose dependencies are satisfied. Learn more about GitLab's advanced pipeline features in the GitLab CI/CD Documentation.
Job Dependencies
# Run jobs in parallel when possible
unit-tests:
stage: test
parallel:
matrix:
- TEST_SUITE: [payments, accounts, users, transactions]
script:
- ./gradlew test --tests "com.bank.${TEST_SUITE}.**" --no-daemon
# Run integration tests after unit tests complete
integration-tests:
stage: test
needs: [unit-tests]
script:
- ./gradlew integrationTest --no-daemon
The example above splits unit tests into four parallel jobs, one for each domain (payments, accounts, users, transactions). If you have four available runners, all four suites execute simultaneously, reducing test time by roughly 75% compared to sequential execution.
Matrix Builds
# Test against multiple Java versions
test-multiple-versions:
stage: test
parallel:
matrix:
- JAVA_VERSION: ['21', '25']
image: eclipse-temurin:${JAVA_VERSION}
script:
- ./gradlew test --no-daemon
Caching Strategy
Caching is one of the most impactful optimizations for pipeline speed. Without caching, every pipeline job would download all dependencies from scratch — for Gradle projects this means re-downloading JARs and re-running dependency resolution on every run.
How GitLab Caching Works
GitLab caches work by compressing specified paths at the end of a job and uploading them to cache storage. Subsequent jobs download and extract these cached files before executing. The cache:key determines cache uniqueness - jobs with the same key share a cache.
Using ${CI_COMMIT_REF_SLUG} as the cache key means each branch gets its own cache. This prevents cache pollution between branches but means the first pipeline run on a new branch must populate the cache. Alternative strategies include using a fixed key (faster but risks cross-branch conflicts) or combining branch and file checksums.
The cache:policy controls whether a job reads from cache (pull), writes to cache (push), or both (pull-push). The build job uses push because it installs dependencies, while test jobs use pull since they only consume existing dependencies. This optimization prevents unnecessary cache uploads.
Gradle Cache
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false -Dorg.gradle.caching=true"
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .gradle/caches
- .gradle/wrapper
policy: pull-push
build:
stage: build
script:
- ./gradlew classes --no-daemon --build-cache
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .gradle/caches
- .gradle/wrapper
policy: push
test:
stage: test
script:
- ./gradlew test --no-daemon --build-cache
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .gradle/caches
- .gradle/wrapper
policy: pull
The GRADLE_OPTS variable disables the Gradle daemon (unnecessary in CI — each job gets a fresh JVM) and enables the Gradle build cache. Caching .gradle/caches stores downloaded dependencies and compiled task outputs; caching .gradle/wrapper avoids re-downloading the Gradle wrapper on every job.
Node.js Cache
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
Quality Gates
Quality gates enforce minimum standards that code must meet before progressing through the pipeline. These automated checks prevent human error and ensure consistency across all code changes, regardless of who submitted them.
Why Quality Gates Matter
Without automated enforcement, quality standards become suggestions. Time pressure, differing interpretations, and human oversight lead to inconsistent application. Quality gates remove subjectivity - code either meets the standard or the pipeline fails.
The specific thresholds (85% coverage, 75% mutation score) represent balance points. Lower thresholds allow too much untested code; higher thresholds face diminishing returns where developers spend disproportionate effort testing trivial code paths. These values have proven effective across many projects. For more details on testing strategies, see Testing Strategy and Mutation Testing.
Coverage Threshold
code-coverage:
stage: quality
script:
- ./gradlew jacocoTestReport --no-daemon
- |
COVERAGE=$(awk -F"," '{ instructions += $4 + $5; covered += $5 } END { print 100*covered/instructions }' build/reports/jacoco/test/jacocoTestReport.csv)
echo "Coverage: $COVERAGE%"
if (( $(echo "$COVERAGE < 85" | bc -l) )); then
echo " Coverage below 85% threshold"
exit 1
fi
echo " Coverage meets threshold"
This job parses JaCoCo's CSV report to calculate exact coverage percentage. The pipeline fails (exit code 1) if coverage drops below 85%, preventing merge. The calculation sums all instructions and covered instructions across all classes, providing an accurate overall metric.
Mutation Testing Threshold
mutation-tests:
stage: quality
script:
- ./gradlew pitest --no-daemon
- |
MUTATION_SCORE=$(grep -oP 'mutation coverage: \K[\d]+' build/reports/pitest/index.html)
if (( $MUTATION_SCORE < 75 )); then
echo " Mutation score $MUTATION_SCORE% below 75% threshold"
exit 1
fi
echo " Mutation score meets threshold"
SonarQube Quality Gate
sonarqube:
stage: quality
script:
- ./gradlew sonar --no-daemon
-Dsonar.projectKey=$CI_PROJECT_NAME
-Dsonar.qualitygate.wait=true
-Dsonar.qualitygate.timeout=300
allow_failure: false
Merge Request Pipelines
MR-Specific Configuration
# Run only on merge requests
mr-validation:
stage: validate
script:
- ./scripts/validate-mr.sh
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
# Check MR title format
validate-mr-title:
stage: validate
script:
- |
if ! echo "$CI_MERGE_REQUEST_TITLE" | grep -E '^(feat|fix|docs|style|refactor|test|chore): .+'; then
BAD: echo " MR title must follow conventional commits format"
exit 1
fi
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
# Validate branch name
validate-branch-name:
stage: validate
script:
- |
if ! echo "$CI_COMMIT_BRANCH" | grep -E '^(feature|bugfix|hotfix)/[A-Z]+-[0-9]+'; then
BAD: echo " Branch name must follow pattern: feature/PROJ-123"
exit 1
fi
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
Artifacts Management
Artifacts are files produced by jobs that need to be passed to subsequent jobs or preserved for later use. Unlike caches (which optimize performance), artifacts are functional requirements - jobs genuinely need these files to operate.
Understanding Artifacts vs Caches
Artifacts and caches serve different purposes. Artifacts contain job outputs (compiled code, test reports, built Docker images) that later stages need. They're guaranteed to be available to downstream jobs. Caches contain dependencies (Gradle caches, npm packages) that speed up builds but aren't strictly required - jobs can function without them by re-downloading dependencies.
The expire_in directive controls how long GitLab retains artifacts. Short expiration (1 week for build artifacts, 30 days for reports) prevents storage bloat. Production-critical artifacts like release builds should have longer retention or be stored in dedicated artifact repositories.
Build Artifacts
build:
stage: build
script:
- ./gradlew bootJar --no-daemon
artifacts:
name: "payment-service-$CI_COMMIT_SHORT_SHA"
paths:
- build/libs/*.jar
- build/classes
expire_in: 1 week
reports:
dotenv: build.env
The name field creates uniquely identifiable artifact archives. Including $CI_COMMIT_SHORT_SHA makes it easy to correlate artifacts with specific commits when debugging issues in deployed environments.
Test Reports
test:
stage: test
script:
- ./gradlew test --no-daemon
artifacts:
when: always
reports:
junit:
- build/test-results/test/TEST-*.xml
coverage_report:
coverage_format: cobertura
path: build/reports/cobertura/coverage.xml
paths:
- build/test-results
- build/reports/jacoco
expire_in: 30 days
Environment-Specific Deployments
Development Environment
deploy-dev:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl config use-context dev-cluster
- kubectl set image deployment/payment-service payment-service=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -n dev
- kubectl rollout status deployment/payment-service -n dev --timeout=5m
environment:
name: development
url: https://dev-payment-service.example.com
on_stop: stop-dev
only:
- main
stop-dev:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl delete deployment payment-service -n dev
environment:
name: development
action: stop
when: manual
Production Deployment with Approval
deploy-prod:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl config use-context prod-cluster
# Blue-green deployment
- kubectl apply -f k8s/deployment-green.yaml
- kubectl wait --for=condition=available deployment/payment-service-green -n prod --timeout=10m
# Switch traffic
- kubectl patch service payment-service -n prod -p '{"spec":{"selector":{"version":"green"}}}'
# Verify
- sleep 60
- ./scripts/smoke-test.sh https://api.example.com/health
environment:
name: production
url: https://api.example.com
when: manual
only:
- tags
before_script:
- echo " Deploying to production - version $CI_COMMIT_TAG"
Security Scans
Security scanning in CI/CD catches vulnerabilities before they reach production. These automated scans detect known vulnerabilities in dependencies, container images, and application code, providing early warning of security risks.
Why Automated Security Scanning Matters
Security vulnerabilities in dependencies are discovered continuously. A library that's safe today might have a critical CVE (Common Vulnerabilities and Exposures) disclosed tomorrow. Manual dependency audits can't keep pace with this constant flux. Automated scanning checks every build against current vulnerability databases, ensuring you're always aware of known risks.
The CVSS (Common Vulnerability Scoring System) provides standardized severity ratings from 0-10. Setting -DfailBuildOnCVSS=7 fails the build for "high" and "critical" vulnerabilities (CVSS ≥ 7.0) while allowing "medium" and "low" severity issues to pass with warnings. This balances security with pragmatism - not every vulnerability warrants blocking a release, especially if workarounds exist or exploitation is unlikely in your context.
For comprehensive security guidance, see Security Best Practices.
Dependency Scanning
dependency-scan:
stage: quality
image: eclipse-temurin:21
script:
- ./gradlew dependencyCheckAnalyze --no-daemon -PowaBackoffEnabled=true
artifacts:
reports:
dependency_scanning: build/reports/dependency-check-report.json
paths:
- build/reports/dependency-check-report.html
when: always
allow_failure: false
The when: always artifact setting ensures reports are generated even if the job fails. This is crucial for security scans - you need to see the vulnerability details regardless of whether the build passed or failed.
Container Scanning
container-scan:
stage: quality
image: aquasec/trivy:latest
script:
- trivy image --severity HIGH,CRITICAL --exit-code 1 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
dependencies:
- build-docker
SAST (Static Application Security Testing)
sast:
stage: quality
image: returntocorp/semgrep:latest
script:
- semgrep --config auto --sarif --output sast-report.sarif .
artifacts:
reports:
sast: sast-report.sarif
Performance Testing in Pipeline
Load Testing
performance-test:
stage: test
image: grafana/k6:latest
services:
- name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
alias: payment-service
script:
- sleep 10 # Wait for service to start
- k6 run --out json=performance-results.json tests/load/payment-test.js
artifacts:
reports:
performance: performance-results.json
when: always
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: manual
Notifications
Slack Notifications
.notify-slack: ¬ify-slack
after_script:
- |
if [ "$CI_JOB_STATUS" == "success" ]; then
COLOR="good"
GOOD: MESSAGE=" Pipeline succeeded"
else
COLOR="danger"
BAD: MESSAGE=" Pipeline failed"
fi
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"$MESSAGE\",\"color\":\"$COLOR\",\"fields\":[{\"title\":\"Project\",\"value\":\"$CI_PROJECT_NAME\"},{\"title\":\"Branch\",\"value\":\"$CI_COMMIT_REF_NAME\"},{\"title\":\"Commit\",\"value\":\"$CI_COMMIT_SHORT_SHA\"}]}" \
$SLACK_WEBHOOK_URL
deploy-prod:
<<: *notify-slack
stage: deploy
# ... deployment configuration
Pipeline Optimization
Pipeline optimization reduces waste - unnecessary pipeline runs consume runner resources, increase costs, and create noise in pipeline histories. Strategic optimization keeps pipelines fast and focused.
Why Skip Duplicate Pipelines
GitLab can trigger pipelines for both branch pushes and merge request creation. Without workflow rules, pushing to a branch with an open merge request creates two identical pipelines - one for the branch, one for the MR. This doubles resource consumption for no benefit.
The workflow rules below implement "merge request pipelines only" for branches with open MRs. When you push to a feature branch with an active MR, only the MR pipeline runs. For branches without MRs (like main), the branch pipeline runs normally.
Skip Duplicate Pipelines
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_BRANCH'
The second rule ($CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS with when: never) prevents branch pipelines when an MR exists. The first and third rules ensure MR pipelines and standalone branch pipelines still run.
Conditional Job Execution
# Run only when Java files change
java-tests:
stage: test
script:
- ./gradlew test --no-daemon
rules:
- changes:
- src/**/*.java
- build.gradle
- settings.gradle
# Run only when frontend files change
frontend-tests:
stage: test
script:
- npm test
rules:
- changes:
- src/**/*.ts
- src/**/*.tsx
- package.json
Debugging Pipelines
Enable Debug Logging
# Set CI_DEBUG_TRACE=true in GitLab CI/CD variables
test:
stage: test
before_script:
- if [ "$CI_DEBUG_TRACE" == "true" ]; then set -x; fi
script:
- ./gradlew test --no-daemon
Retry Failed Jobs
test:
stage: test
script:
- ./gradlew test --no-daemon
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
Further Reading
Internal Documentation
- CI Testing - Testing in CI/CD pipelines
- Branching Strategy - GitFlow workflow
- Pull Requests - GitLab merge request process
- Repository Structure - GitLab repository configuration
External Resources
Summary
Key Takeaways
- Fast feedback loops - PR pipelines complete in < 10 minutes
- Parallel execution - Run independent jobs concurrently
- Quality gates - Enforce coverage (85%+), mutation score (75%+)
- Fail fast - Stop on first critical failure
- Caching - Cache Gradle/npm dependencies for speed
- Artifacts - Build once, deploy everywhere
- Security scans - Dependency scan, container scan, SAST
- Environment-specific - Dev auto-deploy, UAT/Prod manual
- Blue-green deployments - Zero downtime for production
- Audit trail - Track all deployments and approvals
Next Steps: Review CI Testing for detailed testing strategies in pipelines and Repository Structure for GitLab repository configuration.