How to benchmark a web application security scanner?

There is a plethora of web application scanner ; every one of which claims to be better than the other. It is indeed a challenge to differentiate between them. We need to benchmark the application scanner against hard facts and not marketing claims. Below are some of the most critical metrics against which you would like to benchmark web application scanner.

1. What is the rate of false positives?

False Positives are vulnerabilities reported by a tool that don’t actually exist. Any web application scanner will throw some false positives. First we need to understand how false positives are harmful. Even though they don’t apparently seem to be harmful; it costs money to remove them. Imagine a little bit of sand in your food. You can’t eat that food; similarly you can’t send a report with false positives to developers.

Removing false positives from web application scanner reports takes a lot of time. Hence it adds to your man-power cost and of course the drudgery of doing boring work. I have seen so many organization losing people because the work becomes monotonous.

So, you need to check the percentage of false positives reported by the web application scanner. The flip side however is that a web application scanner can minimize its percentage of false positives by limiting its coverage which leads to the next question.

2. How many classes (or percentage) of vulnerabilities does it cover?

False negatives or vulnerabilities missed out is another critical element. You need to understand the percentage coverage of the web application scanner to ensure that critical vulnerabilities are not missed (particularly at the expense of not having to report false positives). You can use WASC 1, WASC 2 or OWASP as a guideline for what should be covered.

3. Which are the classes web application scanner does not cover?

If a web application scanner does not cover certain classes of test (which is always the case), you should know: which are those classes? How important are the classes of test for your business? Can you live without them?

4. How good is the coverage of the crawler? Is there any benchmark?

Crawlers are the fundamental part of any web application scanner. The first step of any testing is crawling. If a page is not crawled then it is not tested. You can benchmark different web application scanner against the number or the percentage of the pages it could crawl. Fast scanning does not mean good scanning. You need a web application scanner which can comprehensively crawl all the pages.

5. How many scans can run in parallel?

Most organizations today have multiple web applications which need to be tested frequently. You need aweb application scanner which can scan multiple tests in parallel. Don’t go by the number stated on the product datasheet but how many it can actually run in parallel without significant degradation of performance. So the best thing is to try it and check this out yourself.

6. How Flexible are the configuration options of the tool?

Does the tool give you the ability to fine tune what test classes it scans for and let you test your production environment safely? Options that allow you to prevent things like automatic form filling, or limiting the number of concurrent threads etc. can prevent unnecessary disruption to your organization when testing your production environment with a tool.

Few more suggestions by readers and community members

Credits:Simon Bennetts, James McGovern, Keighley Peters

How long does it take to run? (Quicker means it could be less comprehensive test. Check for number of tests/hour etc)
How long does it take to learn and configure to work effectively?
How much does it cost?
What are the licensing terms?
How many organizations use the tool? How satisfied are they?
Are there any industry recognition/analysts mentions (e.g. Gartner)?

The selection of appropriate scanner can be very challenging as every organization has developed their applications differently. By considering the metrics discussed above, organizations can benchmark their application scanner to evaluate the effectiveness of a scanner and make a right choice for their organization.

More: Have you nominated yourself for CISO Platform 100 ,2016 ? Click here to nominate

All Articles