Pipeline Patterns

Real CI/CD configurations for TriBITS projects. These patterns work across platforms and scale from single-package builds to multi-repository integration.

CI/CD pipeline flow chart showing build, test, and deploy stages

Why Pipeline Patterns Matter

A build system is only as useful as the CI that runs it. TriBITS provides the structure, but you still need pipelines that use that structure intelligently. The patterns here are distilled from real projects. They are not theoretical ideals.

Every pattern addresses a specific problem: slow builds, flaky tests, cross-repo breakage, or wasted compute. Pick the ones that match your situation.

Pattern 1: Basic PR Pipeline

The simplest useful pipeline. It builds and tests the changed packages plus their dependents.

# Generic CI configuration (adapt to your platform)
steps:
  - name: Checkout
    uses: checkout

  - name: Configure
    run: |
      cmake -D MyProject_ENABLE_${CHANGED_PACKAGE}=ON \
            -D MyProject_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON \
            -D MyProject_ENABLE_TESTS=ON \
            -D MyProject_TEST_CATEGORIES=BASIC \
            -B build

  - name: Build
    run: cmake --build build -j$(nproc)

  - name: Test
    run: cd build && ctest -j$(nproc) --output-on-failure

This pattern keeps PR builds fast because it only touches what changed. The ENABLE_ALL_FORWARD_DEP_PACKAGES flag ensures that downstream breakage is caught.

When to use it

  • Every pull request.
  • Every push to a feature branch.

When it is not enough

  • When you need to validate the full project (use the nightly pattern instead).
  • When changes affect build infrastructure, not just packages.

Pattern 2: Nightly Full Build

A scheduled pipeline that builds everything and runs extended tests.

schedule:
  - cron: "0 2 * * *"

steps:
  - name: Checkout
    uses: checkout

  - name: Configure Full Build
    run: |
      cmake -D MyProject_ENABLE_ALL_PACKAGES=ON \
            -D MyProject_ENABLE_TESTS=ON \
            -D MyProject_TEST_CATEGORIES=NIGHTLY \
            -D CMAKE_BUILD_TYPE=Release \
            -B build

  - name: Build
    run: cmake --build build -j$(nproc)

  - name: Test
    run: cd build && ctest -j$(nproc) --output-on-failure

  - name: Report
    run: cd build && ctest -D NightlySubmit

The nightly build catches problems that incremental builds miss: environment drift, implicit dependencies, and interactions between packages that are rarely changed together.

Pattern 3: Matrix Builds

Test across multiple compilers, build types, and configurations. TriBITS makes this easier because the configure step is parameterized.

strategy:
  matrix:
    compiler: [gcc-12, gcc-13, clang-16]
    build_type: [Debug, Release]
    mpi: [ON, OFF]

steps:
  - name: Configure
    run: |
      cmake -D CMAKE_CXX_COMPILER=${{ matrix.compiler }} \
            -D CMAKE_BUILD_TYPE=${{ matrix.build_type }} \
            -D TPL_ENABLE_MPI=${{ matrix.mpi }} \
            -D MyProject_ENABLE_ALL_PACKAGES=ON \
            -D MyProject_ENABLE_TESTS=ON \
            -B build

Matrix builds are expensive. Run them nightly, not on every PR. Use the PR pipeline for quick feedback and the matrix for thorough validation.

Pattern 4: Multi-Repository Integration

When your project spans multiple Git repositories, you need a pipeline that tests the integration.

steps:
  - name: Checkout Main Repo
    uses: checkout

  - name: Checkout Extra Repos
    run: |
      git clone https://github.com/org/extra-repo-1.git repos/extra1
      git clone https://github.com/org/extra-repo-2.git repos/extra2

  - name: Configure Superbuild
    run: |
      cmake -D MyProject_EXTRA_REPOSITORIES="extra1;extra2" \
            -D MyProject_ENABLE_${CHANGED_PACKAGE}=ON \
            -D MyProject_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON \
            -D MyProject_ENABLE_TESTS=ON \
            -B build

  - name: Build and Test
    run: |
      cmake --build build -j$(nproc)
      cd build && ctest -j$(nproc) --output-on-failure

The critical detail is that the dependency graph spans all repos. A change in extra-repo-1 might require recompiling and testing packages in the main repo.

Pattern 5: Conditional Test Stages

Run BASIC tests first. If they pass, proceed to NIGHTLY tests. This gives fast feedback while still running thorough tests on viable candidates.

steps:
  - name: Configure
    run: cmake -D MyProject_ENABLE_ALL_PACKAGES=ON ...

  - name: Build
    run: cmake --build build -j$(nproc)

  - name: Basic Tests
    run: |
      cd build
      ctest -L BASIC -j$(nproc) --output-on-failure

  - name: Extended Tests
    if: success()
    run: |
      cd build
      ctest -L NIGHTLY -j$(nproc) --output-on-failure

This is a practical compromise. BASIC tests run in minutes and catch most regressions. NIGHTLY tests take longer but only run when the basics pass.

Build Caching Strategies

Compilation caching is essential for keeping CI costs down. For TriBITS projects, the main options are:

ccache: Works well for C++ builds. Set CMAKE_CXX_COMPILER_LAUNCHER=ccache and point the cache at a persistent directory.

Build directory caching: Cache the CMake build directory between CI runs for the same branch. This avoids reconfiguring and recompiling unchanged packages. Be careful to invalidate the cache when the package list or dependencies change.

Container image caching: Pre-build TPL dependencies in a Docker image and use that as the CI base. This avoids recompiling Boost, MPI, or other large dependencies on every run.

Platform-Specific Notes

GitHub Actions

Works well for open-source TriBITS projects. Use matrix strategies for compiler/config variants. The cache action can persist ccache directories between runs.

GitLab CI

Native support for multi-project pipelines, which maps well to multi-repo TriBITS builds. Use artifacts to pass build directories between stages.

Jenkins

Flexible but requires more setup. Use pipeline libraries to share TriBITS-specific build logic across jobs.

Cloud Build Platforms

For teams using cloud-based CI, Google Cloud Build and similar services offer scalable compute. The key is to use build caching aggressively because cloud builds start from scratch each time.

Comparison of pipeline patterns showing tradeoffs between build time and coverage

Monitoring Pipeline Health

A pipeline is only useful if people trust it. Monitor these metrics:

  • Build success rate. Should be above 95% on the main branch. Below that, developers stop trusting CI.
  • Median build time. Track this over time. If it is creeping up, investigate.
  • Flake rate. Percentage of failures that are not caused by code changes. Keep this below 2%.
  • Time to green. How long from push to a passing build. This is the number developers feel.

If any of these metrics degrade, fix the pipeline before adding more features to the project. A broken CI pipeline is worse than no CI because it creates a false sense of security.