SecOps: Integrating security from the start

Roll up, new *Ops term of the year - SecOps. Cynicism aside this is something that is both a passion of mine and a concept I believe will become increasingly more important. The zero-trust security model is starting to gather pace, fast. It is a concept that I have seen in my career for over 10 years (high assurance systems) - it just has finally hit the mainstream!

Billy came to work to start on the Next Big Thing™. Stand up done, no major blockers today. As every other day he logged in to his laptop, to start work on a drag and drop UI for this sprint.  While waiting for VS Code to load, he was busy checking the news when a boobytrapped ad sneaked a backdoor onto his laptop. Anyway, Code loaded up, he got about bashing through this simple bit of code. Library from CDN link pulled, check. Django library to support PDF generation, check. Cooking on gas as they say. Push to Git using SSH and coffee break time.

Back from coffee break time and the tests need writing, CI is passing but test coverage has dropped, as new tests has been written for this new code. Billy gets cracking and by lunch, unit and integration tests are done. BOOM on fire!  Push to CI, coverage is back up, ready to push upstream.

Now this might be an extreme version of a workflow, but the point here is zero trust is about appreciating that the real-world is inherently hostile to software delivery security.

How could this have gone wrong?

An insecure machine allowed a backdoor to take hold, this could be used for future exploits including information leakage, malicious damage, malicious code changes, you name it. Never trust an endpoint.

No mention of using version pinning or hash verification for the libraries that have been pulled into our project. An unpinned / unverified library could easily be compromised and be integrating into Billy's product unnoticed.

Git over SSH is great, it's convenient and fast. But do you have a password on your private key? If someone pwned Billy's laptop, what would they able to push upstream?

There's no mention of a PR for Billy's changes. How is this organisation gating the merge of new code into master and into upstream environments?

Trust nothing, check everything, security should be explicit, not implied.

What does Zero Trust mean in the context of software delivery (as well as infrastructure)? To me it means;

• never assume that your product is secure,
• with agility comes risk for security regression - security review can be slow (but also, rapid change allows us to secure our products quicker and cheaper),
• security mitigations need to start from the moment of code inception (not a final gate, annually,  just before deploying Next Big Thing™)

How would I identify zero trust in the real world? Let's see how Billy from an alternate universe is getting on.

Billy came to work to start on the Next Big Thing™. Stand up done, no major blockers today. As every other day he logged in to his laptop, to start work on a drag and drop UI for this sprint.  While waiting for VS Code to load, he was busy checking the news when a boobytrapped ad attempted to sneak a backdoor onto his laptop. No sweats, AV picked it up. Code loaded up, he got about bashing through this simple bit of code. Library from CDN link pulled,SRI tag applied, check. Django library to support PDF generation, version pinned in pip requirements file, check. Cooking on gas as they say. Push to Git using SSH (after tapping his YubiKey) and coffee break time.

Back from coffee break time and the tests need writing, CI is passing but test coverage has dropped, as new tests has been written for this new code. Billy gets cracking and by lunch, unit and integration tests are done. BOOM on fire!  Push to CI, coverage is back up, ready to request a merge of the PR.

It's the same theme, but many steps have an additional protection in place to embed security best practice into the developer workflow from day 1.

Like the fiction approach to learning DevOps? I highly recommend Gene Kim et al's 'The Phoenix Project' and their upcoming sequel - 'The Unicorn Project'.

OK, so embedding security into the developer's workflow, check.

What about your CI pipeline? That's going to depend on a few factors, including your CI platform, threat landscape and deployment platform. A few things to consider;

• Static Application Scanning Tools (SAST) - These will inspect code at a filesystem level (i.e docker container, deployment bundle) for known issues, primarily around OS, application and library bugs that are identified via fingerprinting i.e out of date docker base image or an old pinned Python library. This is pretty deterministic and is pretty much solely dependant on up-to-date CVE databases.
• Dynamic Application Scanning Tools (DAST) - Designed to scan your application in it's operational state, the sum is greater than its parts etcetera.... This will perform 'attacks' against your application using known techniques, for example XSS or searching for incorrectly set cookies. This type of scanning is more of an art, as you are emulating what a skilled hacker would be doing. Different products are likely to be a better fit for different types of application compared to others.
• Signing - this has been a requirement in some industries for many years, but code should  be signed to ensure integrity (and with sufficient infrastructure, authenticity).
• Security isn't just about resilience to malicious intrusion, it also relates to integrity and availability. Good development hygiene is important to ensure that these security risks are mitigated  - ensure that linting and test coverage is run in your pipeline. Easy to read and well maintained code is likely to result in less risky changes, ensuring high levels of availability for your solution. Code that is easily reviewed and thoroughly tested is more likely to perform what is supposed to do, even in failure scenarios, ensuring integrity.

A few select solutions that have taken my interest;

GitLab

With either an opensource project or for Private project on Ultimate / Gold tier, you get out of the box DAST, SAST, Container Scanning and Dependency Scanning. All you need to do is include the appropriate template into your CI definition and the jobs are run automatically. Results are piped back into the CI results overview to then allow triage. I have duplicated OWASP's Juice Shop Demo Webapp over on GitLab. Checkout the Security Tab on the pipeline run to see this deep integration. If you're already on GitLab CI this is a no-brainer.

Google Cloud Build

If your target deployment platform is GKE on GCP, then Google Cloud Platform could provide some real value. It supports the ability to sign your images as part of the build process. GKE supports Binary Authorization which acts as a gate keeper for new images being pushed into the cluster. Multiple assertions can be used by Binary Authorization, with a healthy range of integrations into other platforms including TwistLock and CloudBees. Fundamentally, this loosely coupled flow allows multiple flows to provide a gate for a container image before it is deployed (for example, scan container, tests pass, human checkpoint).

This is a more extensible version of the Content Trust feature in Docker Enterprise - where only signed images will be run by the Docker Engine. If you're not using k8s, then this might be an option for you.  

Roll your own

There is certainly scope for DIY solutions here, albeit with some hard work being required. Self hosted or cloud CI you have options including;

• Using clair container scanning to check for out of date OS and application dependencies,
• Integrate OWASP ZAP into your CI workflow (Jenkins or custom integration with Glue),
• Scan your application using OpenVAS (a popular DAST tool),
• Integrate one of the defacto tools for static analysis (both code quality and security) - SonarQube.

What custom solution you select, and how you run it, will depend on your commit cadence (security scanning runs can be quite lengthy) and what you are looking to mitigate.  

This is a massive topic and is rapidly growing. As this is more around technology integration, deep dive into your CI platform (Jenkins, Teamcity, GitLab, CircleCI) and see what you have included out of the box, and what you can integrate yourself, you might be surprised!

As ever, thoughts and comments always welcome below or contact me directly - tom at fluffycloudsandlines.blog.