Simplifying Infrastructure - Vanilla CI/CD for complex infrastructure (Part 2)

How Bruce Lee would build deployment pipelines that flow like water

All fixed set patterns are incapable of adaptability or pliability. The truth is outside of all fixed patterns.

— Bruce Lee

Bruce Lee sits quietly watching an engineering team struggle with their infrastructure deployment. They have Atlantis servers to maintain, Terragrunt configurations that span dozens of files, and a deployment process that requires three different tools just to get a simple network change into production. After observing their frustration for an hour, he speaks softly:

"Empty your cup so that it may be filled. You have filled your deployment pipeline with so many tools that there is no room for the infrastructure itself. Be like water - flow directly to your goal without unnecessary obstacles."

Bruce Lee - “Be like water - flow directly to your goal without unnecessary obstacles.”

This four-part series breaks down how I build Terraform CI/CD for complex shared infrastructure, in a simple way, that works in the real world:

• Part 1 exposes why your current tooling is probably fighting against you:

• Part 2 (current post) demonstrates how vanilla CI/CD pipelines can handle the most complex infrastructure deployments without specialised tools. You'll discover the universal patterns that make any infrastructure deployable with nothing more than standard GitLab CI or GitHub Actions, plus the Terraform stack organisation that scales from simple resources to enterprise-wide platforms;

• Part 3 looks into branching strategies, and explores my personal preference for GitLab Flow for complex infrastructure, while allowing developer teams to use whichever strategy they prefer;

• Part 4 shows you the functional composed-repository pattern that scales from startup to enterprise without the complexity death spiral;

Main Insights

• Vanilla pipelines handle complexity in a simple way: Standard CI/CD platforms can deploy any infrastructure without additional orchestration tools;

• Functional areas create natural deployment boundaries: Organising by business function minimises change conflicts and coordination overhead;

• Universal Terraform patterns enable consistency: Standardised stack structure means every infrastructure component deploys the same way;

• Simplicity scales better than sophistication: Water-like pipelines adapt to requirements; rigid frameworks break under unexpected loads;

• Platform choice is personal preference: Both GitLab CI and GitHub Actions can handle enterprise infrastructure when used with vanilla approaches (I personally prefer GitLab CI to GitHub Actions)

The vanilla philosophy: Maximum power, minimal complexity

Bruce Lee’s martial arts philosophy centered on “the way of no way“ - using techniques so fundamental they could adapt to any situation without requiring specialised forms. The same principle applies to infrastructure deployment. The most powerful CI/CD approach isn’t the most sophisticated - it’s the one that flows naturally with your infrastructure’s actual shape.

Vanilla CI/CD means using GitLab CI or GitHub Actions in their pure form. No external orchestration frameworks. No specialised infrastructure deployment platforms. No complex dependency management systems. Just clean, readable pipelines that deploy infrastructure the way infrastructure actually should be deployed. This is about recognising that the complexity should live in your infrastructure definition, not in the deployment mechanism.

This article is very AWS centric, but the same principles extrapolate to different software and infrastructure providers.

The universal deployment pattern

The foundation of vanilla infrastructure deployment is a Terraform stack pattern that can deploy anything consistently. Based on real-world implementations that handle everything from simple S3 buckets to complex multi-region networks, here's the structure that makes vanilla pipelines possible:

terraform-stack/
├── config.tf                 # Backend and deployment configuration
├── providers.tf              # Cloud provider setup with role assumption
├── variables.tf              # All input parameters with validation
├── outputs.tf                # Exposed values for other stacks
├── remote_state_imports.tf   # Dependencies on other stack outputs
├── main.tf                   # The actual infrastructure logic (multiple files)
└── environments/
    ├── dev.tfvars            # Development environment parameters
    ├── staging.tfvars        # Staging environment parameters  
    └── prod.tfvars           # Production environment parameters

The magic happens in config.tf, which creates a universal deployment interface:

# Universal backend - works with any state management
terraform {
  backend "s3" {}
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Universal deployment parameters
variable "target_account_id" {
  description = "AWS account for deployment"
  type        = string
}

variable "deployment_role" {
  description = "IAM role for deployment operations"
  type        = string
}

variable "backend_config_bucket" {
  description = "S3 bucket for Terraform state"
  type        = string
}

variable "backend_config_region" {
  description = "Region for state management"
  type        = string
}

# Universal provider configuration
provider "aws" {
  region = var.aws_region
  assume_role {
    role_arn = "arn:aws:iam::${var.target_account_id}:role/${var.deployment_role}"
  }
}

This pattern means every infrastructure component - whether it's deploying a complete AWS Landing Zone or a single Lambda function - uses exactly the same deployment interface. Your CI/CD pipeline doesn't need to understand what infrastructure it's deploying; it just needs to know how to execute the universal deployment pattern.

Functional areas: The natural deployment boundaries

Real-world infrastructure has natural boundaries where related components change together, but unrelated systems evolve independently. Organising your infrastructure around these functional areas creates atomic deployment units that eliminate change conflicts.

stacks/
├── control-tower/             # Landing zone foundation
│   ├── .gitlab-ci.yml         # Deployment pipeline for this area
│   ├── landing-zone/          # Core Control Tower setup
│   └── controls/              # Organizational controls
├── networking/                # Network infrastructure
│   ├── .gitlab-ci.yml         # Network deployment pipeline
│   ├── core-network/          # Core networking fabric
│   ├── firewall-rules/        # Security rules and policies
│   └── dns-zones/             # DNS management
├── iam/                       # Identity and access management
│   ├── .gitlab-ci.yml         # IAM deployment pipeline
│   ├── emergency-access/      # Break-glass procedures
│   └── central-roles/         # Cross-account role management
└── platform-services/         # Shared platform capabilities
    ├── .gitlab-ci.yml         # Platform deployment pipeline
    ├── logging/               # Centralized logging infrastructure
    ├── monitoring/            # Observability platform
    └── security-scanning/     # Security tooling deployment

Each functional area contains its own deployment orchestrator pipeline that knows how to deploy all infrastructure within that domain. This creates natural boundaries where networking changes don't require coordination with logging deployments, and platform updates don't interfere with tenant provisioning.

The key insight is that functional areas map to how your organisation actually operates. Your network team owns the networking area, your security team owns the security tooling area, and your platform team owns the shared services area. Each team can deploy their infrastructure independently while maintaining integration through well-defined interfaces.

The universal pipeline pattern

Here's how any functional area deploys, regardless of complexity. This exact pattern works for simple storage buckets and complex multi-region platforms:

# .gitlab-ci.yml for any functional area
stages:
  - validate
  - plan
  - deploy

variables:
  TF_VERSION: "1.10.0"
  TF_ROOT: "${CI_PROJECT_DIR}/${STACK_PATH}"

before_script:
  - cd $TF_ROOT
  - terraform init
      -backend-config="bucket=$TF_BUCKET_NAME"
      -backend-config="region=$TF_DDB_REGION"
      -backend-config="key=$TARGET_ACCOUNT_ID/$CI_PROJECT_NAME/$STACK_NAME.tfstate"
      -backend-config="dynamodb_table=$TF_DDB_NAME"
      -backend-config="encrypt=true"

validate_infrastructure:
  stage: validate
  script:
    - terraform validate
    - terraform fmt -check
    - checkov --directory . --quiet --exit-code-non-zero

plan_deployment:
  stage: plan
  script:
    - terraform plan
        -var="target_account_id=$TARGET_ACCOUNT_ID"
        -var="deployment_role=$DEPLOYMENT_ROLE_NAME"
        -var="backend_config_bucket=$TF_BUCKET_NAME"
        -var="backend_config_region=$TF_DDB_REGION"
        -var-file="environments/${ENVIRONMENT}.tfvars"
        -out=tf.plan
  artifacts:
    paths:
      - $TF_ROOT/tf.plan
    expire_in: 2 hours

deploy_infrastructure:
  stage: deploy
  script:
    - terraform apply tf.plan
  dependencies:
    - plan_deployment
  when: manual
  only:
    - main

This pipeline deploys anything. Whether you're provisioning a simple S3 bucket or a complex multi-account networking fabric, the deployment process remains identical. The complexity lives in the Terraform code where it belongs, not in the deployment orchestration.

Master pipeline orchestration

The root repository contains a master pipeline that orchestrates functional area deployments with natural dependency management:

# Root .gitlab-ci.yml - coordinates all infrastructure deployment
stages:
  - foundation
  - networking  
  - platforms
  - tenants

# Deploy foundation infrastructure first
control_tower_deployment:
  stage: foundation
  trigger:
    project: stacks/control-tower
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $MASTER_ACCOUNT_ID

iam_management:
  stage: foundation
  trigger:
    project: stacks/iam
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $MANAGEMENT_ACCOUNT_ID

# Deploy networking after foundation exists
network_infrastructure:
  stage: networking
  trigger:
    project: stacks/networking
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $NETWORK_ACCOUNT_ID

# Deploy platform services after networking is ready
platform_services:
  stage: platforms
  trigger:
    project: stacks/platform-services
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $PLATFORM_ACCOUNT_ID

logging_infrastructure:
  stage: platforms
  trigger:
    project: stacks/logging
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $LOGGING_ACCOUNT_ID

# Deploy tenant management after platforms are ready
tenant_provisioning:
  stage: tenants
  trigger:
    project: stacks/tenant-management
    strategy: depend
  variables:
    ENVIRONMENT: $CI_COMMIT_REF_NAME
    TARGET_ACCOUNT_ID: $MASTER_ACCOUNT_ID

GitLab's stage-based execution naturally handles dependencies. All foundation infrastructure is completed before networking begins. All networking completes before platform services deploy. The dependency management is built into the pipeline structure—no external orchestration tools required.

This master pipeline can coordinate dozens of functional areas across hundreds of AWS accounts without additional complexity. Each functional area evolves independently, but the integration happens through well-defined stages that mirror how infrastructure components depend on other infrastructure components.

Why I prefer GitLab CI

While both GitLab and GitHub can handle complex infrastructure deployment with vanilla approaches, GitLab CI feels more natural for infrastructure work. This is a personal preference based on my years of building and deploying complex shared infrastructure, not an endorsement related to technical superiority.

• Sequential execution of sub-pipelines: Infrastructure has natural dependencies. In AWS, your VPC must exist before your applications. Your Transit Gateway must be deployed before you can attach VPCs. Your landing zone must be configured before you can provision accounts. GitLab's stage-based approach with the ability to invoke sub-pipelines mirrors this sequential reality;

• Variable Inheritance: GitLab automatically passes variables from master pipelines to sub-pipelines, reducing configuration overhead. When you're orchestrating infrastructure across multiple functional areas, this eliminates the need to explicitly pass every configuration parameter;

• Easy job templating without losing execution context: Whereas GitHub allows you to define re-usable actions where context is lost, GitLab has powerful templating YAML capabilities that allow standardisation while maintaining environment context simplicity;

• Integrated Security: GitLab includes container scanning, dependency scanning, and secret detection without additional configuration. For infrastructure deployment pipelines that need to meet enterprise security requirements, this integrated approach reduces integration complexity;

But honestly, these are preferences. GitHub Actions can handle the same complexity with slightly different patterns. The key insight is using vanilla approaches on either platform rather than adding specialised infrastructure deployment frameworks.

GitHub Actions: The alternative approach

For teams that prefer GitHub's ecosystem, the same vanilla patterns work with workflow orchestration:

# .github/workflows/infrastructure.yml
name: Infrastructure Deployment

on:
  push:
    branches: [main, development]
  workflow_dispatch:

jobs:
  foundation:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        functional-area: [control-tower, iam]
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Foundation Infrastructure
        uses: ./.github/workflows/terraform-deploy.yml
        with:
          functional-area: ${{ matrix.functional-area }}
          environment: ${{ github.ref_name }}
          target-account: ${{ secrets.FOUNDATION_ACCOUNT_ID }}

  networking:
    needs: foundation
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Network Infrastructure
        uses: ./.github/workflows/terraform-deploy.yml  
        with:
          functional-area: networking
          environment: ${{ github.ref_name }}
          target-account: ${{ secrets.NETWORK_ACCOUNT_ID }}

  platforms:
    needs: networking
    runs-on: ubuntu-latest
    strategy:
      matrix:
        functional-area: [platform-services, logging]
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Platform Infrastructure
        uses: ./.github/workflows/terraform-deploy.yml
        with:
          functional-area: ${{ matrix.functional-area }}
          environment: ${{ github.ref_name }}
          target-account: ${{ secrets.PLATFORM_ACCOUNT_ID }}

The reusable workflow terraform-deploy.yml contains the universal deployment pattern, eliminating repetition while maintaining flexibility. GitHub's approach requires more explicit configuration but provides clearer interfaces between components.

The advantage of simplicity

Vanilla infrastructure pipelines are not the most sophisticated deployment approach, but they adapt to any infrastructure requirement without breaking. They flow around obstacles that would shatter rigid frameworks. They persist through organisational changes, technology migrations, and evolving requirements.

In summary, here’s what you get when you keep your CI/CD pipelines simple:

• Learning Investment: Engineers familiar with GitLab or GitHub contribute immediately. No specialised training on framework-specific concepts or debugging techniques;

• Maintenance Overhead: No additional tools to update, configure, or debug. Your pipeline maintenance effort scales with your infrastructure, not with framework complexity;

• Vendor Independence: Pure Terraform / OpenTofu plus standard CI/CD can migrate between platforms. No lock-in to specialised deployment frameworks that may become obsolete;

• Debugging Efficiency: Standard pipeline logs and Terraform errors have extensive documentation and community knowledge. Framework-specific issues require specialised expertise;

• Hiring Simplicity: Finding engineers experienced with GitLab or GitHub is straightforward. Finding experts in specialised infrastructure frameworks can be expensive and time-consuming;

• Total cost of ownership for vanilla approaches scales linearly with infrastructure complexity. Specialised framework costs can behave randomly as edge cases accumulate and framework limitations require workarounds.

Next up: Part 3 discusses how using GitLab Flow branching strategy helps to keep complex infrastructure manageable, testable, how to deploy and promote across environments, while minimising risk (your developers can continue to use whichever branching strategy they want).

I invite you to subscribe to my blog, and to read a few of my favourite case-studies describing how some of my clients achieved success in their high-stakes technology projects, using the very same approach described.

Have a great day!

João

Visit my website | Linkedin profile