Return on security investment

Soon I will begin a research project into quantifying and expressing return on security investment. From conversations I've had with many conference attendees, there's a need for developing a basic understanding of how to measure ROSI so that budget money for security magically becomes unlocked. I plan to assemble a presentation on this for 2006's events.

If any of you have personal thoughts on ROSI, or some tips that work for you, please comment here or email me (steve.riley@microsoft.com). I'd love to include your ideas. Thanks!

Published 03 January 06 10:40 by Steve Riley

Comments

# Alun Jones said on January 3, 2006 5:52 PM:
Performance review time already, Steve? :-)
# Phil Jackson said on January 3, 2006 9:52 PM:
I think you are trying to do the impossible. I will be surprised if you could pull of such a document. I mean what is your companies reputation worth if you have to disclose that you have compromised a lot of personal information? Just look at the company cardsystems that exposed 40m credit card numbers, they lost their relationships with the credit card companies. As you all have said security is spending a lot of money so nothing happens. Either your boss and/or company understands that or they do not. I have worked for companies that understand this and those that don't and I think the only way to try and convince those that don't is with LOTS and LOTS of FUD (fear uncertainty, and doubt).

Usually I try to stay away from FUD because I think IT guys use it too often, but in the case of security it's the only way to convince management of the need for budget dollars. Lots of examples with the idea "you don't want this to happen to you do you?" Here's how we can prevent it. I like Jesper's comment that of Security, usable, and cheap you can pick two.

So my advice include in your presentation lots of stories and examples and make some up if you can't find them ;-)
# Alun Jones said on January 4, 2006 11:26 AM:
My earlier comment notwithstanding (implying that Steve might be looking to find ways to describe the return on his employer's investment in him), there's importance to evaluating ROSI.

Many companies are loath to fund anything that they cannot pitch to the shareholders as being directly or indirectly relating to the building of value for the company.

Currently, security projects get funded for a couple of reasons - regulation, and "oh my god, look at how badly our stock went down when we screwed up security".

Regulation is often just a case of matching check-lists, and with legislation taking years to get drafted and implemented, the regulated security measures are never timely, they are simply the minimum you could have reasonably expected a company to implement five years ago.

The reactive support for security is the same - instead of being a push to provide better future security, it's generally enough for companies to prevent the re-occurrence of the event that just passed - anything more than that, even if it's to prevent a predictable future event, is underfunded.

What I think Steve is looking for is any measure whereby a security manager can say something like "if I buy this $100 piece of software, I will prevent a $100,000 cost that would occur if we are hacked, which we estimate will happen with a 5% probability."

Those kind of measures can exist; if a similarly-sized company spends a certain amount of money clearing up a virus infection, that your virus scanner protected your company from, that's a return on security investment. If fraud used to be a loss equal to 3% of your revenue, but now has dropped to 1% because of security measures, that's a return on security investment.

I think those figures are out there, but they're going to be very hard to come by, and some may be rather nebulous (for instance, the frequent "we lost $X bn to software piracy" which always generates the argument of whether those who pirated the software would have ever bought it).
# Stephen Moore said on January 4, 2006 2:13 PM:
How much does a virus cost? Or compromised customer information? How much is your company's reputation worth? Maybe you can find dome figures out there, but I think at the end of the day it's back to lies, damn lies, and statistics -- and that's not much different than spreading the FUD that a previous poster suggested.
# Alun Jones said on January 4, 2006 5:30 PM:
So what, then, do you base your business decisions on? You can either come up with numbers that have some bearing on reality, but may be significantly wrong, or you can just throw darts at a board. I'd choose the first, because you can tweak the model when you find out where it doesn't match. Ignoring Lewis Carroll, a watch that loses a minute a day is more use to me than one that is stopped.
# Stephen Moore said on January 4, 2006 8:03 PM:
Why, we just buy whatever the vendors try to sell us of course!

To figure ROSI, as you've already stated, you need to know the A) your cost, B) the cost of bad event, and C) the probability of bad event happening if you don't spend the money. (A more thorough look would also include other factors like the probability of bad events happening if you don't spend the money -- it's probably not 0 -- and the lost productivity from implementing the security measure in question.)

My contention is that you probably don't know B and you definitely don't know C. And a difference between .01% probability and .02% probability is equivalent to doubling the cost of the product. To make up numbers and then pretend that you have the magic ROSI is dishonest. And it can cost you a lot of time and money to come up with your numbers.

I think a more reasonable approach is to spend your money to mititgate the most likely threats.
# n00dles said on January 5, 2006 4:42 AM:
Yeh I gotta say that's a gargantuan task Steve... the last poster (Stephen Moore) pretty much hit the nail on the head.

Last place I worked got absolutely hammered by blaster. That was enough to kick the business into (re)action. After that, we got a proper patch management system for the server infrastructure and NEVER had a problem with getting outages for applying patches. That company had over 60,000 employees. There isn't a risk-based business case in the world that could've got the same result. A catastrophic event was necessary.

Steve, if I may be so bold as to suggest, perhaps a better approach would how to best prepare the proposal for new security measures for use after a catastrophe. In my experience, that's the crucial time to get such a proposal in. If you're too busy dealing with a technical post mortem instead of proposing measures to stop something similar from happening in the future, you may lose the best opportunity you will ever have to get the security budget you want.

It's unfortunate, but in my experience that's just the way it is.
# Alun Jones said on January 5, 2006 11:04 AM:
If ROSI is the wrong answer, then at least let's look at the question (at least, the question I see as being behind this research topic):

How do corporate entities (or interested home users?) turn security from a reactive (*) process to a proactive (**) one?

(*)quick, our network has been taken over by a virus, now what do we buy to clean it up and prevent that particular virus from happening again; lather, rinse, repeat
(**)this product is designed to protect against an attack that probably hasn't happened to us yet - while we have the time to make a reasoned decision, is it worth buying this?
# n0one said on January 5, 2006 12:03 PM:
"Last word in" post(*)

(*)Alun impression

IT staff makes the most use of themselves creating and later dismissing/solving FUD problems. It takes some effort to create easily solved problems that still require a few weeks/months of "implementation." But usually, that's easier that actually doing your job. Besides, IT staff exist in the background unless they are fixing something. You can't get a raise if you don't get noticed.

Security is really the best target for FUD too. Reboot a few servers in the middle of the day, spend a few more days in the server room. Then proclaim that the hackers have been expelled and that while working on the system you found 3 more holes the last IT guy left open.


Just reply to this thread if you need any more tips.
# Pete said on January 6, 2006 12:21 AM:
Hi, Steve - Check out my blog at http://spiresecurity.typepad.com/spire_security_viewpoint/ for some details on ROSI and ROI.
# Stephen Moore said on January 6, 2006 8:57 AM:
Pete, I crown you the smartest guy that's replied to this topic. But I still don't understand/agree with your approach to calculating the probability of something bad happening.

I think your premise that "We know when a compromise occurs because it is self-defining" is flawed. Let's say a user's password is compromised. Certainly we can audit successful and unsuccessful login attempts. But how will we know if an unauthorized person logged in, unless they start doing other obvious damage? Will you be able to detect data theft? And how do you proactively (please don't flame more for that, n0one) calculate probabilities of things haven't to you yet?

If you estimate something as 2 in a million and it's really 1 in a million, it's still the equivalent of the cost of the security vendor charging twice as much for their product. And a 100% increase or more in cost makes it difficult to even rank competing security measures by ROSI, never mind figuring out with certainty what it is.

I think we've got to get these probabilities as accurate as a weatherman predicting rain before ROSI can become a useful tool.
# n0one said on January 6, 2006 9:37 AM:
Which isn't that difficult thanks to http://www.noaa.gov/rosi
# Chris said on January 9, 2006 5:06 PM:
Actually all of this sounds like Steve is sliding to the right along that horrible line and becoming Marketing
# Andrew said on January 15, 2006 9:02 PM:
Ideally I would like to see if the ROSI curve is linear or logarithmic? (I know easier said than done)

What are the sweet spots (or high return areas) for ROSI? Does a $100,000 spent dillegently on (and updating) security awareness posters have a higher return than a $100,000 spent on a mail filter upgrade?

How does measuring a ROSI metric change over time? For example try to apply these metrics to computer security 10 years ago:
http://csrc.nist.gov/publications/history/

As much as I hate to say it, budget approval for me is based on tangible results. If my CFO can hold it in his hand or chart it in excel, my chance of bugeting is very good. Unfortunately the opposite applies.

Steve, I welcome any tools like this. On a lighter note, case studies a few 1000 miles off the USA Conetinental shelf are always good too.
# Michael said on January 16, 2006 1:26 PM:
Steve,

I invite you to review the following methodology that we use to address part of the ROSI dilemma: http://www.css-security.com/downloads/security_kaizen_faq.pdf

I hope it helps with your research.
# Alex said on February 23, 2006 7:50 AM:
Hi,
I yust read the thread and like to state the following:
Some of you have correctly stated, that it is quite hard to predict on future harms and what they might cost you.
In my oppinion, there is a whole business making money with predictions like that: insurance companies. They predict what it will cost to fix all damaged cars in the next year an charge me money for that. And in the end of the day, the insurance earned loads of money.  I think, if we want to get close to a satisfying accurate ROSI, we should take a look to what insurances do to predict future damages and what they might cost you.
New Comments to this post are disabled
Page view tracker