What do you think of when you hear the term ‘information security’ or even the term ‘cybersecurity’? If you think about how it all works, you may think about vulnerabilities, firewalls, intrusion detection systems, anti-virus or perhaps something else entirely. What probably doesn’t come to mind are terms like metrics and measurement. These are elements of information security that seems to get short shrift a lot, in spite of their importance. No matter what aspect of a company’s security posture, metrics are essential.
As an example, if we start on the low end of the scale, the one where everyone has an impact on the company, we talk about security awareness. Companies today generally have security awareness programs to help their employees know how to do the right things when it comes to interactions outside the company and especially with corporate resources and sensitive information. These awareness programs often start with some form of training — computer-based or video. The question is: how do we measure the effectiveness of this training? Ultimately, what is the goal of a security awareness program? To make sure employees know what information security is and the impact they have on it? Of course not. It’s to ensure that employees alter their behavior in order to better protect the organization and its resources. So how do you measure behavior change?
Metrics aren’t always about numbers. Sometimes we just need help retracing the steps after something has happened (e.g. qualitative information). Recently, I was looking at trying to measure some behaviors with respect to firewalls – asking: what happened and when did it happen? What I discovered was that neither iptables nor firewalls, the two Linux-based firewalls, provided any persistent details when rules were changed or what the rule changes were. It doesn’t appear as though it’s possible to even turn on that level of logging. One open source firewall where a Web interface is used to make changes is PFSense. This is a firewall based in the BSD operating system. After making changes to the rules, there was no indication of a change having been made in any of the log files. How do we measure over time the changes to rulesets and the impact they have had if there is no record of the changes to begin with?
Measurement comes down to identifying the problem, much like many other aspects of information technology (or even other industries and endeavors). In the first case above, what is the problem? The problem is that humans can have a negative impact on the security posture of an organization. So, what are we measuring? Are we measuring whether we’ve trained all the people in the organization? We could, and it’s easy to measure that, but what would be the purpose? If your organization has to demonstrate compliance to a set of standards, this may be useful. It’s more important to measure behavior, and more importantly, changes in behavior as a result of training.
One way to measure behavior changes, is to send e-mails with links that should look like they are untrustworthy. If the links are clicked (the URL would be one that goes to a site controlled by information security or information technology), there is evidence that the behavior hasn’t changed. What do you do with the information when behavior hasn’t changed? Put people through the training again? If it didn’t work the first time, what would suggest that it may work the second time?
And this is why measurement is important. Without this data, you don’t know when something is going wrong. You also won’t know what is going wrong. Unfortunately, there are no easy resolutions. More data isn’t necessarily better. The best approach when it comes to measuring security is to clearly identify the problem or situation. It’s essential to take a logical and rational approach to this and not feel like you have to protect against absolutely everything. Once you have identified the situation, you can determine what you need to measure, as in the case of awareness training. The really hard part is in interpreting the data. In the case of security awareness, we know that people are not making decisions based on the training they have had. Do you address that by sending the people through training again? Do you re-evaluate the training?
It’s not always easy to make the right decisions but having the right data to inform your decision is essential. You can only have that if you think through ahead of time what the right data is, so you can ensure you are collecting it.