If you are security professional, or simply interested in securing applications, you are likely to have come across the term “shift left”. The phrase is coined from the idea that application security is more effective (and significantly cheaper) when embedded early in the development lifecycle. And it’s a topic I presented to delegates at Information Security World in London this week.

Many studies have, in fact, proven that fixing vulnerabilities during the design stage is as much as 100 times cheaper than fixing them once the application is in production*. The same is true for operations. Being able to correctly identify and fix infrastructure defects prior to or during deployment can greatly mitigate and control abuse cases.

However, in the age of DevOps, where new ideas do not last long and competition is fierce, companies must move quickly to meet customer demands. The distinction between builders and operators continues to blur. The builders are now in control of their own destiny as technologies like containers and Infrastructure-as-a-Service (IaaS) allow developers and operators to collaborate tightly to control how applications are packaged, deployed and – of course – secured.

With freedom and almost unlimited flexibility come potential pitfalls. All too often, functionality and business agility trump due diligence. The most basic security considerations and controls are neglected in favor of speed and functionality. As DevOps greatly reduce the time it takes to go from idea to a finished product, traditional application security practices struggle to keep up. 

Additionally, DevOps now need to care about infrastructure security, traditionally a remit of operations teams.  A poorly configured S3 bucket with an overly permissive configuration can seriously undermine efforts to secure the application.

Securing DevOps

So, how do we go on about securing DevOps?    

The DevOps community has embraced the “shift left” mantra with a passion and several books, talks and projects have been dedicated to this very quest – mostly with excellent results. This has resulted in many security vendors offering fully programmable security products (e.g. Dynamic Application Security Testing, Static Application Security Testing , Web Application Firewalls) where automation and tight Continuous Integration and Continuous Delivery (CICD) is achieved through APIs.

Nevertheless, while early visibility is great and allows organizations to catch and deal with security issues early in the development lifecycle, it is really all about prevention. 

At some stage, an organization may need to respond.

So is security facing a losing battle when it comes to DevOps? Not necessarily. Automation and API driven security controls can play a big part in securing DevOps practices that produce apps at breakneck speed. That speed of change can also serve to allow for the remediation of issues far quicker than in traditional development lifecycle projects that must ‘wait’ for the opportunity to remediate.

To understand how this can work in the real world let’s take a fictitious start up business called ABC as a case study.

The interesting case of ABC

ABC is a start-up that has developed a killer app called “where you are”. Based on user profiles and common interests, the app puts in touch like-minded people within a few miles radius. To scale quickly and increase market share, ABC has adopted a cloud-first business strategy opting to leverage DevOps practices and public cloud. For this, it has chosen Amazon Web Services (AWS).

Architecturally, the app is quite simple. It has a front end that runs on mobile or web. An application tier that looks up users coordinates via GPS and matches compatible profiles. User and application data is stored on an AWS Relational Database Service.

The app runs within a Docker container on Elastic Beanstalk, and builds its immutable IaaS through Cloud Formation scripts.

However simple the above setup may seems, it is in fact ridden with security vulnerabilities.

  • Ops have no visibility of whether the cloud infrastructure meets compliance requirements or AWS best practices. Therefore, they may miss configurations and vulnerabilities introduced  by incorrectly coded Infrastructure-as-Code (IaC).
  • Lack of conformance between multiple accounts, services and regions may lead to the disparate security policies, which are difficult to maintain and enforce, potentially introducing configuration issues exploitable by an attacker to gain access.
  • Poor visibility of the application health and its threat landscape prevent accurate threat modelling and the creation of accurate abuse cases thus affecting its design.
  • No ability to automate remediation of infrastructure defects or virtually patch vulnerable code greatly increases the risk exposure of the application. 

So, what would be the impact of a breach on ABC? What would the damage be?

As a worst-case scenario, an attacker could exfiltrate confidential Personally Identifiable Information (PII) for the application’s users. This information could include names and addresses, including email addresses, and credit card details, potentially leading to a much bigger fallout with serious ramifications. The reputational damage would be significant and ABC would likely incur legal and restore and recovery costs.

The breach

It is nowadays common practice for organizations to separate their development stage from production. This is a sensible approach as it allows DevOps teams to work in the safety of a sandboxed environment and fix bugs before code is released into production. 

Frequently however, Dev environments are not subjected the same level of scrutiny and rigor as production. Security is rarely considered at this stage as DevOps teams’ primary objective is to “just get it working”. It is often assumed that, once in production, the services will be adequately secured.

This causes a proliferation of security bad practices ranging from shoddy code to overly permissive Identity Access Management (IAM) roles used in an attempt to get the app working and ready for staging.

In the case of ABC, a member of the DevOps team happens to have hard coded AWS secrets in a YAML configuration file instead of using environment variables to dynamically retrieve these. The latter being the recommended best practice.

The YAML file was then uploaded to an S3 bucket to automate its retrieval as part of the IaaS build process.

Several free tools are available to scan the internet for open S3 buckets. Through such a tool, an attacker was able to retrieve AWS credentials and log into ABC’s Dev environment.

Once inside, the attacker started to move laterally looking for unpatched EC2 instances. After an initial recon, the attacker finds a build server vulnerable to an exploit that allows them to escalate privileges and push a code commit to the code repository master branch.

Without automated code scanning, the attacker was able to introduce a stored XSS vulnerability into the application code base.

The application is now vulnerable to an XSS attack, which the attacker uses to steal authentication cookies from legitimate users. The scene is set for exfiltration of sensitive user data from an RDS database.

Granted, many things have to go wrong for such an attack to succeed and result in a breach. But these scenarios are not uncommon and are definitely on the rise

The aftermath

So now, ABC is hacked but it takes the company six days to discover the newly introduced vulnerability as part of a routine penetration test. To make things worse, it takes developers two more days to fix the code and release a patched version of the app. 

ABC’s application has now been vulnerable for eight days, potentially leading to thousands of users’ accounts being compromised and their data stolen.

To add insult to injury, ABC does not know exactly how many accounts have been hacked because it does not actively monitor application access through a Web Application Firewall.

Lessons learnt

As always, hindsight is a beautiful thing but there are some basic precautions that ABC could have taken to prevent such a devastating breach that ABC should learn from this:

  • An AWS security scan/audit would have found over-permissive roles on the AWS S3 buckets, thus preventing AWS credential being leaked in the first place.
  • Automated Dynamic Security testing embedded in CICD pipeline would have found XSS vulnerabilities straight away, enabling developers to fail the build and prevent the release of insecure code in production
  • Application monitoring tools would have alerted security operations of the XSS attack in flight and would have allowed ABC to patch the app while Dev fixed the code, massively reducing the amount of time the app was vulnerable for.

In essence, a combination of good baseline security practices and automation would have prevented the breach or, in a worst case scenario, allowed remediation within hours rather than days.

Prevent and respond 

This fictitious case study helps highlight some of common pitfalls DevOps team fall into in an attempt to expedite application development and deployment.

It illustrates how early visibility of security issues is essential to prevent breaches but makes a strong case for being able to react and respond quickly when breaches do occur.

While no security expert can ever guarantee an application will never be hacked, by combining prevention with the ability to respond quickly to security incidents, it is possible to greatly reduce an organization’s risk exposure to external and internal threat.

Best of all, security no longer needs to be perceived as a blocker to business agility but rather as a facilitator that enables organizations to take controlled risks and move quickly in an ever increasingly competitive market.


*IBM System Science Institute Relative Cost of Fixing Defects