Implementing Data Center Consolidation

Cindy Cassill is the director of systems integration in the office of the CIO for the US Department of State. Prior to her current position at the Department of State, Cindy has over 30 years of federal IT experience.  She was the CIO at the FAA Regions & Centers.  She also was the CIO at the US Army Test and Evaluation Command and was the director of IT at the deputy assistant secretary of the army for civilian personnel.   This article highlights portions of Cindy Cassill’s presentation and the steps the agency took for their consolidation. Click Here to download the entire presentation and transcript at length.   Planning: So we knew looking ahead that these old datacenters, you know, they weren’t scalable, they weren’t Tier 3 and they weren’t gonna carry us…We put together our plan and then we took it to our leadership. And we knew we had to get this leadership support before we could go any further…We brought in a company, they helped us do a business case analysis. And we looked at all the datacenters that we had across the department… we did a survey and we found out that we had over 77,000 feet of computing space that if we really condensed and consolidated and racked things property, we would only need 10,000 square feet of datacenter space. So when we took this to the undersecretary for management and said, you know, we’re using expensive office space in Washington D.C. for datacenters…we were out of power-space and cooling and we were gonna save him money. We walked in with a four, sort of pronged plan for an immediate datacenter, and then also a redundant datacenter in the future and then also how we would do our development networks and also how we would handle our top secret networks. Funding: So we go to the funding people and say, okay, the undersecretary said we’re gonna do this…about six or seven months or so down the line, the whole stimulus funding issue comes up in government. And our undersecretary said, you know, I think we need to get RF funding and we need to build the second datacenter now while we’re still building out the first datacenter. Now we are going to have a chargeback model… there may be fights yet to come, but for right now, it’s kind of – I hate to say this, but a little bit of bait and switch thing.  You know, we get them in and then we worry about the funding later.  But it does make it easier and it makes it more efficient to consolidate that way.   Building the right team is really very important: I walked in and had a datacenter manager that had great vision and he still has great vision.  And he has great technical skills to make things happen.  And what we decided was that we would separate him out from the operations part of it… the two of them together work so hand-in-hand – and it couldn’t work any other way. Having knowledgeable project managers who can go out to the customers and talk to them about what they need, and plus have the technical skills to be able to do the virtualization and to have the NetApp engineers and that sort of thing, has been really great.   Communicating the vision: Telling our customers what we’re doing was also, and still is, a vital part of our process… we put together a briefing. Make sure that your customers know you have the right connectivity when you’re putting datacenters on the west coast, not on the west coast, but on the western power grid…taking the eastern datacenter is away from Washington. We funded three full-time people to go work on the network side to support our project so that we made sure that we got the right attention that we needed to do the job.   Service model: We’re trying to keep it as simple as we can.  And we’re primarily focused on co-location.  But with co-location, we do the monitoring and we do the backups and we help them migrate to the datacenter.  But we’re trying to keep that to a minimum so that it doesn’t distract us from our datacenter consolidation. Virtualization is key: What we tell our customers is that we will help you assess whether a system needs to be – can be virtualized or not, because everybody thinks theirs can’t be virtualized.  You know, I’m gonna suffer in performance and space and all that sort of thing.  And then also you have to look at their life-cycle times.  If they’ve just bought new life cycle equipment, they’re not gonna want to virtualize. Right now we buy virtual infrastructure.  If you virtualize, you come on our platform, if you want physical machines, you have to bring your own physical machines to the datacenter.  And then a lot of people are doing – if they stay with physical machines they’re doing it on a life cycle basis as well.  Nobody’s moving old equipment around and that’s a good thing. But the benefits of the virtualization…is that it is centrally funded, it reduces the acquisition time, because we’ve already bought that infrastructure, so we can just migrate them over and bring them in.  It gives you high availability.   Key stakeholder buy-in: Don’t forget who your stakeholders are, and that you have to coordinate and coordinate and coordinate with them.  And all the way from the vendor, if you’ve leased the facility like we have in one datacenter to the facilities managers if you have other datacenters that the government owns, your security. As far as your customers are concerned, they’re datacenter owners right now.  So you have to coordinate and try to figure out how you’re going to close those rooms down and how you’re going to have them demolished and repurposed as office space.  We deal a lot with people who just don’t want to give up that space.   Project management: We developed three different kinds of plans.  We developed our overall datacenter transition plan, which gave us our view of all the things that we needed to do for the datacenter consolidation.  We also shared this with our customers.  Then we also had our customer transition plans.  And then we have the individual systems plan. You have to make sure that when you’re getting ready to migrate a system, especially to a new datacenter, that you’ve already migrated all the infrastructure that supports that and that you’ve migrated everything that that system owner owes that they think they know about to get that system over that, because there’s so many moving pieces and parts that they don’t always know all of those if the documentation isn’t there. Moving our DMZ infrastructure and our Internet connections and that sort of thing, that has taken a long time for the east, and we’re gonna go through that for the west again.  But don’t underestimate the length of time that it takes to do that.