The perception of continuous security testing vs the reality

‍

Continuous security testing is becoming increasingly popular in organisations with a frequent rate of development. The perception of continuous testing is attaining coverage of all assets across entire environments, 365 days of the year. However, there’s a grey area around what is perceived to be true continuous testing, and continuous testing capabilities.

‍

Full coverage, i.e. testing for vulnerabilities consistently across an entire estate 100% of the time, is an unrealistic expectation. And so the reality of what continuous testing looks like for a typical organisation can be quite different. 

‍

The precursor to continuous testing

‍

Looking at the precursor to continuous testing, the need for it can be originated in two areas. 

‍

Baseline penetration testing

‍

Baseline penetration testing consists of a comprehensive test once a year. But this is often not enough for many organisations who need vulnerabilities to be identified proactively. In most organisations, product releases happen more than once a year. And so therefore, vulnerabilities are created more than once a year. Moreover, attacks don’t wait around for penetration tests and so waiting for one annual baseline test no longer cuts it. 

‍

This stimulates organisations to adopt internal testing processes. Using automation tools, regular scanning processes are installed which begins the move towards more frequent security testing. However, more frequent, in this case, still doesn’t mean continuous. And there is a limit to the types of vulnerabilities these tools are able to find that only human tests can accomplish. 

‍

Bug bounty programmes

‍

This encourages organisations to turn to bug bounty programmes. Here they open up to the community and invite them to try and compromise their systems, theoretically, every minute of every hour of every day. 

‍

However, just like frequent internal scanning processes, it’s difficult to get assurance that everything is being covered all of the time.

‍

Bug bounties are an important part of a security testing program. However, it still doesn’t quite hit the nail on the head when it comes to continuous testing, or at least the industry's interpretation of continuous testing.

‍

Naturally, bug bounty is limited in its focus on the external attack surface. But, a good portion of the security testing that businesses do isn't just external. Scanning at code level, pen tests where credentials are given and internal networks are all examples not typically covered in attack surface management. 

‍

Of course there are bug bounties where the wealth of information for the test provided is more generous than others. But the main limitation in preventing it from being a source of continuous testing is the assurance of the level of coverage. This not only includes conventional coverage of all 20 features in one app, but also coverage across the whole organisation. When you leave bug bounty companies to their own devices, often you are at the behest of the researchers themselves. Getting direction or commitment from the researchers are common challenges with any bug bounty programme.

‍

In comparison, continuous testing is generally a strategic conversation about all of the assets the company owns. It's about striking a balance across the entire organisation instead of focusing on a particular area. 

‍

Combining approaches to achieve continuous testing

‍

The next question to unpick is, can companies combine multiple testing methods to achieve continuous testing? As mentioned, bug bounties are often applied in certain areas of a business. Therefore, can automated scanning tools be applied across the remaining areas where human testing is not necessary? Ultimately, this is what continuous testing should be about. Understanding which areas of a business can be covered by a combination of SAST and DAST, and where penetration testing or bug bounties can be folded in at different points to complete the coverage. 

‍

Furthermore, there are now much more sophisticated automated testing tools that create valuable continuous-like results. ASPM, for example, in application security, is able to orchestrate specific scanning against specific endpoints when changes are made. Its ability to deconstruct an application, understand when a new API endpoint is touched, a new form field is introduced or when a new feature is deployed provides an element of continuous coverage. However, without a complete record of testing, blindspots are still common when understanding what vulnerabilities might be present but haven't been detected by the tool.

‍

Although this isn’t necessarily a limitation of the tool itself, more a consideration for the depth of understanding needed on what the tool isn’t capable of doing. 

‍

Ultimately, however, combining these methods still purely increases the frequency of tests. To become truly continuous, there’s another element to consider.

‍

Where, what and how to test

‍

Increasing the frequency of tests doesn’t address the challenge that comes with testing everything. Of course there are methods that are quicker than others. DAST, for example, is a way of automating penetration testing activity. But it can still be slow, loud, and responsible for false positives and red herrings. So we know that simply testing everything all of the time is not only impossible, but ineffective.

‍

And so understanding where to test, what to test and how to test it is the crucial part of a continuous testing strategy. The instinct is to start where vulnerabilities are likely to be - the environments changing the most, the most complex features and functionality or the most recent deployments.

‍

From there vulnerabilities can be addressed at the time of development change and intelligently assessed on the likelihood of vulnerabilities introduced. Changes at code level hold the key to finding new vulnerabilities and the tool to help finding them. If, for example, it’s likely the change will have introduced a business logic flaw, we know that running an automated scan won’t pick this up. But a manual test will.  Analysing code changes in this way is the key to avoiding wasted blanket testing and reducing the chance of vulnerabilities slipping through the net.

‍

Testing for vulnerabilities at the time of change also makes vulnerability remediation much faster and simpler for the developers as they’re likely to still be familiar with the original code.

‍

So to answer the key question above: Where to test? At code level. What to test? Code changes. How to test? The most appropriate tool for the vulnerability likely to be introduced. 

‍

The challenges to achieving continuous testing

‍

Industry standards

‍

Within the industry there are a few forms of security testing that have become a safe bet. An annual penetration test, for example, generally satisfies an auditors questions around security testing. And for companies where cyber security isn’t at the forefront of their activities, this method ticks a box. 

‍

With legislations like DORA and FedRAMP however, there is a push towards more continuous and change based testing. They raise the issue that companies can no longer  rely on an annual assessment and material changes inside your business need to be assessed throughout the year. 

‍

But the standard that has been here for decades has been an annual assessment perhaps supplemented with regular scanning in the meantime. And we as an industry still have work to do to change this.

‍

Capacity

‍

Another challenge is simply capacity to work strategically vs fighting fires. Fixing some of the fundamental flaws in a security testing programme would class as a strategic action. But how much time do security teams have to think strategically vs fighting fires? For most companies, the reality is the latter. And so taking the time to implement a new strategic process, even if it will reduce the number of fires in the long run, often gets deprioritised. 

‍

Tool selection

‍

Often being able to efficiently and effectively determine when to use the right tool or person can take a certain level of knowledge or expertise within a security team. Although there are ways of automating the classification of vulnerabilities likely to be introduced by a specific change, this can take time and practice to implement successfully.

‍

Budget

‍

Another consideration is cost. A misconception is that continuous security testing is more affordable. Of course cost is a perception and is entirely based on an individual business. If, for example, a company reduces outsourced consultancy to only test what needs to be tested, there would likely be an overall cost reduction. But at the same time, the outcome of continuous testing is a greater level of assurance, which generally comes at a slightly higher price.

‍

There’s also the consideration that moving into continuous testing results in a continuous discovery of vulnerabilities.This in itself can incur a cost as the rate of vulnerability remediation will alter. If teams are used to only fixing vulnerabilities once a year or at the end of a month, there will need to be an adjustment of response to accommodate this change.

‍

Quality of data

‍

In order to accurately analyse what and how to test, the quality of data in largely influences the quality of data out. So the more visibility over estates and attack surfaces, activity within an environment, existing processes and changes being made is going to make a big difference to the testing strategy. This information should also guide the tools and technologies that make up a stack. There isn't a one size fits all, or a scanner that is better than another. There are simply scanners that are suited to certain vulnerabilities and vice versa. Similarly, there isn’t a standard amount of manual testing that is required, but this should be individual to each company. Understanding these variables and building a programme that works best for the individual entity can sometimes be seen as a challenge in itself.

‍

Missing context

‍

A crucial element to successful testing, continuous or not, is the context provided around vulnerabilities and the environment they’re found in.

‍

For example, security teams often already have a wealth of information that’s valuable to a pentester before they start. Information like the kinds of vulnerabilities that typically exist within their system, the vulnerabilities that tend to be introduced from their code languages and architecture, or simply a log of changes that have taken place since the last annual assessment. 

‍

These pieces of context help to make a usually quite short engagement where a tester is looking through a large variety of different vulnerabilities, a much more effective process. It’s commonly known that context is extremely valuable in finding and remediating vulnerabilities quickly, but often it’s a lack of awareness, or perhaps oversight, that can often make testing processes harder than they need to be. 

Summary

‍

Continuous testing optimises the capability of discovering vulnerabilities. But it's not about covering every system every minute of every day. It's about being pragmatic about the systems and tools available and the most effective way to orchestrate those tools to find vulnerabilities as efficiently as possible. And this starts with understanding changes made at code level.

‍

Instead of taking a catch-all method of layering testing methods on top of each other, it focuses on thinking strategically about what to do and when. It provides rich context on the vulnerability found and the environment it’s found in so vulnerabilities can be found and remediated quickly and efficiently.

‍

If you’re intrigued about implementing continuous testing and wondering where to start, check out our Continuous Testing Orchestration Platform for more information.

‍