Case Study

CI/CD at Scale: Automating the Path from Code to Production

Founding the shared pipeline catalog, designing instanced pipeline architecture, and encoding separation of duties into a CI platform serving 400+ engineers.

The Challenge

With 400+ engineers across many product teams, HMH had chosen Concourse CI as the deliberate alternative to per-team Jenkins sprawl: one centralized build platform, each team scoped to its own namespace. But a CI platform is only as good as its shared building blocks. Without a common catalog of well-built pipeline resources, every team reinvents secrets handling, image publishing, and notifications, and quality varies with whoever wrote it last.

The Approach

I founded the organization's shared Concourse resource catalog and stayed its largest contributor for four years. The first pull request set the standards every consuming team still follows: development guidelines for authoring a resource, a consistent build convention, a PR template, and semantic versioning. The catalog grew to include:

A Vault resource that lets pipelines pull secrets safely, written carefully so secret material never leaks into build logs
Adaptive kubectl and helm images that target multiple Kubernetes versions from one image, which quietly eased every fleet-wide EKS upgrade
Team notifications, artifact repository integration, and code-quality gate resources, adopted from the community and hardened for internal use

For the ML platform I designed the standardized instanced-pipeline architecture: a meta-pipeline that sets itself GitOps-style, then fans out per-tenant, per-environment child pipelines from reusable templates covering artifact builds, image mirroring and promotion, model approval, and scheduled training runs.

Separation of duties is encoded in the platform itself. Credentials for lower environments can promote builds through development, integration, and certification, but never to production. Production credentials can only make that final hop. Every build gates through static quality checks before anything publishes. No single credential can walk code from a laptop to production.

Alongside this, I led the GitHub Enterprise migration from self-hosted to cloud with comprehensive rollback procedures and zero disruption, updating the automation that every pipeline depended on.

The Impact

400+

engineers served by one CI platform

contributor to the shared resource catalog I founded

4 years

of catalog stewardship, standards still in use

Zero

disruption through the GitHub Enterprise migration

The catalog became how teams build pipelines at HMH, and the instanced-pipeline architecture became the CI backbone of the ML platform. I tested every example before publishing the documentation, so it matches real CLI behavior.

Lessons Learned

Paved roads beat mandates. Teams adopted the shared catalog because it was the easiest correct path. Standards live longest when they ship with working examples. The guidelines from that first pull request are still followed years later because they came with a build convention and a template anyone could copy.

Splitting promotion authority across teams and environments was the decision that made the pipeline a security boundary. It cost a little convenience up front and bought a lot of trust. The pipeline is part of the platform's security model, and building it that way from the start was far cheaper than retrofitting it later.