Software Build Pipeline = The Finest Hacker Target

The heart of the CI/CD Build Pipeline is the Build Scheduler. Usually, in an automated fashion, the Build Scheduler collects the source code, compiles it on build nodes, packages binary artifacts, performs automated testing and scanning,  uploads binary artifacts to binary repositories, triggers automated deployments, reports back the status of a build to the source code management tool, etc. In order to meet these needs, the Build Scheduler requires privileged access to all components of the build landscape. While compromising source code or binary repositories would allow malicious modification of the build outcome, unauthorized access to the Build Scheduler would often yield compromise of the security of all components. Therefore, we need to put the Build Scheduler in the center of our security assessments.

Challenges

Organisational Challenges

Organisationally those CI/CD build pipelines are often operated by the development teams themselves and in some cases are part of the productive environment of the operated service (devops). In any case, these landscapes sit between the development and operations/delivery organizations often with no clearly defined lines of responsibilities. There might be the IT department, which provides infrastructure to development, but operational responsibility above the OS would be usually with the development team.

Vulnerable by Design

Build systems are vulnerable to code execution vulnerabilities by design. In a build system, source code gets compiled to binary often with high privileges during compile time, meaning any developer can potentially hack the build system. But can we trust all developers? Can we design our build system in a way that even if a malicious developer pushes code, we have the right controls and build processes in place to mitigate such an attack?  This is by far the most difficult technical challenge that a build system needs to overcome and mitigate.

Build System Evolution

Development organizations do not plan for a full-blown, full-fledged all-inclusive build environment from the beginning. The build landscape and its functionality grows and evolves over time with the delivered products. At the beginning the software build may be the part-time responsibility of one developer. Over time this may evolve to a team of ten, with 24/7 support and a release schedule for the build system itself.

All Custom 

Software build pipeline teams rely on common off-the-shelf software like Github, Jenkins, Artifactory, etc., but the implemented processes and automation are all custom to the project and the developer's needs. Build teams are the kings of automation/scripting and it is often the case that automation and security are difficult to bring together (keyword: credentials).

Operations and Development

Nowadays we embrace DevOps as a development and operations model, but with this assumption we imply that a developer is automatically a network engineer, a hypersacler (AWS/Azure/GCP) architect, an OS/database administrator, virtualization and storage administrator at the same time... and let's not forget - a security expert. While this may very well work in multi-disciplined and balanced teams, it does not have to be always the case.

Historically priorities of operations and development have been very different and the ask which we have here as security officers is that development (where usually build systems reside) do secure and proper system operations.

Build Systems are Difficult to Audit

Build pipelines implement complex and custom processes. It requires understanding of software development processes which need to be validated on technology level.  Little understood and unknown terms like artifacts, branching, branching of feature branches, bugfix branching, merging, merging to main code line, forking, voting, product release management, product maintenance life cycles, etc. scare auditors and pentesters and they often revert back to known common technology ground like network and OS security which leaves a huge blank spot on the security map of build landscapes, namely the development and build processes.

Build Systems Are Fun!

Security of CI/CD pipelines is a challenging and rewarding task. Securing an inherently vulnerable environment is the security expert's challenge. In the build teams of your organization, you will find and work with incredibly skilled technology experts who care about their build landscape and know all nuts and bolts of their complex environment. They do not take criticism lightly, but presenting the right arguments will result in swift actions towards improvement, because they care.

Our Service

We have deep understanding and expertise in the field of Build landscapes. We know the best practices and the operational difficulties of pipeline operations teams, which helps us provide you with viable improvement recommendations. 

We perform security assessments for CI/CD build pipelines by organizing workshop sessions with build pipeline operations teams and development process stakeholders in order to get an understanding about the common IT processes mentioned above. We discuss the development and release management processes and map those to the technology layer. Through the workshop sessions we gain valuable knowledge for the next phase where we perform a penetration test on the CI/CD Build pipeline. We document any identified issues and provide mitigation recommendations  to improve the security state of the pipeline.

You need support securing your build systems? Contact Us!  We help clients achieve secure and reliable builds by ensuring the security of the Continuous Integration/Continuous Delivery Build Pipelines.