The IT Trench Warfare Blog ...Tips, Tricks, & References to Simplify your life in the IT Trenches

are not dirty words!!

Change Management is one of those phrases that immediately gets a negative reaction.  For many IT professionals change management brings to mind a life-stealing process that turns the simplest little modifications into major pilgrimages that require authorization from every person in the organization, several external entities, and a couple of supreme beings as well. And certainly there are some organizations that have perverted the concepts of change management into some sort of change-denial process in a misguided attempt to achieve stability.

First lets set the record straight, Change Management is not about preventing changes.  The goal of change management is really to promote an environment of controlled (read documented) changes that have undergone review where necessary to ensure that one fix doesn’t break five more things.  It is critical that change management allows needed changes while enforcing accountability, repeatability, and report-ability. The stability, and recoverability of our systems require us to know what changes are being made  or not, and system stability and availability are our primary concern as a system/network administrator.

So how do we design change management processes for our organizations that meet these goals:

• Be Accountable
• Do it the Same Everywhere
• Write that Down

The first goal of change management is increasing accountability and this is easily accomplished by simply put a procedure in place that requires that all changes (no matter how small) must be authorized by someone in a position of authority.  Somebody has to step up ad take the heat or praise for OK’ing the  change.  Now the key to not letting change management run amok is to keep the scope of the changes match the scope of required authority.  In other words, downloading definition updates for the antivirus on a server does not require the authority of the CEO, CFO, COO, or really any member of the executive management team.  The server admin or network manager should be able to make that call.  And by the same token you should not let the server admin authorize an OS upgrade without a change of that scope going through a significant review to discover and mitigate any potential issues prior to the implementation of the change.  Scoping the changes is really key to insuring that small changes are not held up needlessly.  Many organizations have very formalized Change Review Boards or something like them that must meet and review all changes and those formalized processes can work quite well as long as we remember scope and have procedures in place to address when the full board has to review a change and when only certain relevant subsets of the board are required. 

Our second goal is to ensure that changes (like client configuration or security) are repeatable.  Many of the smaller changes that we make regularly are designed to correct application issues, OS hangs, browser security, and the like and it is vital that once we diagnose, implement, and test a solution that we be able to deploy that exact identical change to many other systems.  There are lots of tools to help us repeat those changes in an automated fashion but what we are really concerned most with is not the speed of dissemination but insuring that every system get the same changes.  The slippery slope here is that without automation tools like scripts, the steps of the change may not be replicated identically on each system and is even more likely as the length of the change process increases.  That makes our third goal so important to insure that our changes are standardized and deployed identically.

The third goal of our change management process is designed to insure that all changes are documented.  Documentation provides the the audit record to follow up on accountability and repeatability, and may even be a compliance requirement depending on our organizational needs.  Two of the most important reasons for documentation include recovery operations and emergency changes.  If we need to recovery all or part of our infrastructure, it is vital that we we have up to date information about the state of our systems and that state information may not always be stored in the form a of a backup. Maybe I’m just paranoid but I like to have full written documentation as well as electronic backups when I need to do a recovery so that even if there is a problem with my backup, I can still rebuild the environment. And it would be entirely unrealistic to expect that every change that ever happens will go through the change management or change review process.  Certainly we should always plan to adhere to the change management policy but also plan for those circumstances that will arise where changes made need to be made outside of the standardized process.  These events should be few and far between but it is vital that we have an exceptions process to insure that when an exceptional circumstance does occur that someone is held accountable for having made that decision and that the change that was made is recorded in the event that we need to roll it back.  Lastly we need to have of that information recorded so that problems that might not arise for several days as a result of that change can be tracked back to the change and reversed as necessary.

So you can see that the change management process for your organization will be designed around your culture, size, and business requirements and be entirely unique to your organization.  That being said, while your change management policy will be unique it should be composed of selections of standard components such as change documentation procedures, change review process, exceptions procedures, and change management forms/requests.  Standard elements selected and arranged based on your individual needs.

So change management doesn’t have to be a change killing process but if properly designed and implemented serves as a process to monitor and record change.  Alright now…

Go Out and Make Some Changes.

Microsoft TechNet has a great article on the 10 Immutable Laws of Security and of all the truisms presented there the most important of all of them is briefly mentioned on the 10th Law.

“security is journey, not a destination”

Secure is not some Utopian locale that you can arrive at by implementing security policies and following best practices.  Secure is an unachievable ideal that we must strive for but know that we can never realize. So that leads us to the idea that security is a process, and that process is really Risk Management. The Institute for Security and Open Methodologies (ISECOM) in the OSSTMM 3 defines security as "a form of protection where a separation is created between the assets and the threat. This includes but is not limited to the elimination of either the asset or the threat." and risk management is essentially the process of evaluating and then mitigating the threats to assets.

Now every security professional knows that information security is really an exercise in risk management, but there is a world of difference in knowing and practicing as evidenced by the daily news of security breeches around the globe.  So why is there this disconnect between knowledge and implementation?  I believe that many well meaning non-security IT professionals just plain struggle with putting risk management concepts into actions, and procedures.  Policies are written by the security professionals, but the procedures to carry forward and enforce those policies are often written and managed by system administrators, network administrators, network managers and the like.  So in a blog about making the day-to-day experience of a administrator a little easier, why would I write an entry on security?  It’s simple really, while security issues may not pop-up every day their arrival however infrequent is sure to make your day much longer and more stressful.

So lets look at the risk management process and how it applies to our day-to-day operations.

First we have to figure out what assets we have and what it would cost us to lose them.  Some things like servers, switches, desks, etc. are pretty easy enumerate and assign dollar values but we also have to calculate the value of our data stores and other ethereal resources such reputation and brand value.  There are several ways to establish a valuation chart for our assets but 2 of the most popular are dollar values and relative valuation.  Relative valuation involves assigning values from 1 to 100 or 1000 for assets with the highest valued assets being assigned the highest numbers.  I personally favor a hybrid system utilizing both relative and absolute dollar value buy first ranking assets by relative value and then filling in dollar values for known assets and finally assigning dollar values for the less obvious items based on their relative rank.  We will use these values a little bit later.

Second we have to identify what threats or risks exist to our assets such as losses due to malware infestation, equipment failures, accidents, catastrophes, and even malicious attacks.  This is easily the most involved part of the process as we have to document every potential loss no matter how small or large.  I suggest breaking common threats such as viral infestation into several scoped entries such as Major, Minor, and inconsequential infestations as the potential for loss varies based on the scope of the incident.

The third part of our process is to determine what vulnerabilities might exist in our systems and policies that increase our risk of loss.  Vulnerabilities come in many disguises including mis-configuration, default settings, missing patches, poor policies, and many others. We must also identify exposures to risk such as allowing access to confidential data.  Now obviously we have to be able to use our data but its very availability is a risk to that data.  Now I am not suggesting that we not make data available but merely that we document risk and later will determine which risks we can mitigate and which one we just have to live with and even more importantly how we can potential reduce the surface area of those exposures that we do have to accept.  A great example would be to make the data accessible but control carefully who it should be accessible to an under what circumstances.

Prioritizing threats is our fourth step and where we finally start putting all the pieces together.  In order to recognize which threats represent the greatest risks to our businesses we have to plug all this information that we have gathered into a simple formula:

Single Loss Expectancy (SLE) * Annual Rate of Occurrence (ARO) = Annual Loss Expectancy (ALE)

Single Loss Expectancy represents the potential dollar loss for an incident.  Now most incidents do not result in total loss but rather a partial loss of value or revenue and you doo need to think not just about direct losses but also indirect losses such as lost revenue during a recovery, additional labor cost for said recovery, unrealized revenue, and other associated costs.  You can get some excellent averaged statistics from several sources such as those listed below:

The Internet Crime Complaint Center

The SANS Institute

IBM X-Force Security Research

These same sources will also give you information regarding the Annual Rate of Occurrence of these type of incidents. Note that some of these items will have an ARO of less than one per year and so will be represented by decimal values less than one.

Finally, buy multiplying the SLE and ARO you will arrive at the expected Annual Loss and armed with that information you will be able to prioritize threats and determine the effectiveness of your mitigation efforts.

You are now ready to develop and implement mitigation strategies to reduce your risk of loss. By evaluating the potential for loss versus the cost the reduce or eliminate that potential, you can make informed decisions about how to best allocate your limited resources in IT Security in the most effective applications.  And by repeating this kind of analysis each year you can develop trend data that will show the true effectiveness of your mitigation strategies. 

Proper application of these strategies will show an ROI for your Security Dollars and go a long way in showing that IT can be a profit center for your organization instead of a cost center.  It will also cut down the number of daily fires that you have to deal with and reduce the instance of those major catastrophes that occur in every organization.

So go manage some risks.