Hello!  My name is Vinnie Liu.  I am a BlueHat speaker, and the Managing Director at Stach & Liu, a security consulting firm whose primary practice area includes helping organizations establish effective application security programs.  A key component of every application security program is the use of tools and experts.  In this post, we discuss the relative strengths and weaknesses between tools and experts, and by doing so, we also learn how these software security resources are best applied in an organization looking to become more proactive with their secure software development lifecycle.


As an organization builds their software security program, they will inevitably find themselves asking the question, “do I hire a consultant or do I buy a tool?”  Unfortunately, the simple act of asking this question is limiting – it poses a false dilemma as the two options need not be mutually exclusive.  Our recommendation, and what we’ve seen from our smartest clients, is the use of both tools and experts.  To understand why, we need to look at what we’re up against and what we’ve got going for us.


Let’s start by saying that we want to assess an application that we (magically) know contains three vulnerabilities.  In our example, we’ve plagued our application with the following:

1. Directory indexing has been enabled.

2. SQL injection exists within the password reset workflow process.

3. A password reset design issue exists.


A couple more pieces of information are needed here.  First, the last two vulnerabilities occur on the same page, and this page cannot be reached without first passing through a previous series of pages.  Second, there are several design issues in the password reset page, and we’ll leave it as an exercise for the reader to identify the password reset design problems.
























When examining our available resources, we can break down our first category, tools, into two varieties – inspection tools and functional testing tools.  Inspection tools perform software inspection at the source code level, e.g. Ounce and Fortify, while functional testing tools perform application review against a live production site, e.g. WebInspect and AppScan.  Our second category is the human brain, which we won’t be breaking down in this article.  So how well do these three tools perform when trying to identify our three security issues?




















 Application Functional Testing Tool – The functional testing tool will quickly and reliably identify the directory indexing issue.  However, because the SQL injection vulnerability requires the application state be at a certain point in order for the issue to expose itself, the scanner would likely have difficulty identifying this issue on its own.  The functional testing tool would never detect the password reset design issue.


Software Inspection Tool – Our static analysis tool will be able to quickly identify the data flow path from source to sink for the SQL injection vulnerability despite it being embedded several pages deep into the application.  However, it would never detect either the directory indexing or design issue.


Expert Software Analysis – An expert analyst would be able to detect all three of these issues, and she should be able to detect the software design issue immediately upon seeing it.  Unfortunately, the time required for the analyst to reliably identify the directory indexing issue would be several times more than that required by our functional testing tool.  The same time issue would apply for our SQL injection issue.  Though manually tracing through the code from sources to sinks is possible, it would be extremely time consuming – with expert analysis, time is money, so this approach becomes financially infeasible.


If we look at how these vulnerabilities can occur in our software development lifecycle, we see that they are introduced in three distinct areas – essentially whenever something is being built or created.  Combining our knowledge of where an issue manifests itself with the tool best-suited to identify it, we notice the following:
























Broadly, we can conclude that functional testing tools are best used to identify misconfiguration problems and perform brute-force tasks like testing for directory indexing, unnecessary files, and so on.  These tools can reduce the amount of time necessary to perform an assessment by taking care of the tedious but necessary iterative tasks.  Inspection tools are best used to build dataflow representation of the software and to identify certain vulnerabilities, which tend to be issues such as SQL injection and XSS, whose root cause is input validation.  Expert analysis is well-suited for identification of design level issues.  In addition, expert analysis is a must for validating the results from the functional testing and inspection tools.  This is because automated tools can generate errors – false positives and false negatives, and by design these tools can never be free of errors (a discussion far longer than this one). 


Our expert is also necessary because she can synthesize the software’s intended functionality and contextual information – a process which the current tools are not capable of performing.  Yet despite the necessity of the expert, using purely expert approach is infeasible because experts take longer at many tasks than automated tools do.  Like digging a well with a toothpick, you shouldn’t be paying a consultant to guess file names in a browser window or to manually recreate dataflow diagrams of an application.  Instead, tools should be used to automate these repetitive tasks.


By analyzing the relative strengths and weaknesses of tools and experts, we come to the simple realization that yes, you should use both types of tools alongside expert analysis in order to develop an “effective” software security program.  You’ll need a functional testing tool, but don’t expect it to find issues in areas that it’s not designed to find.  You’ll need an inspection tool as well.  You need to have an expert in the loop, but apply where needed and not in excess. 


The right tools, the right way, at the right time.