Tuesday, September 15, 2015

Report: FCC Saved IT Upgrade From Jaws Of Defeat

The government gets a bad rap for failing to meet deadlines and messing up major system overhauls.

But, according to Federal News Radio,  it’s rare we get insight into how one agency pulls a potential major blunder from the jaws of failure. But that’s what happened to the Federal Communications Commission last week during a major systems upgrade.

The FCC kicked-off its modernization effort Sept. 2, shifting more than 200 servers and transferring more 400 applications associated with those servers to a commercial cloud.

But nearly a week later, the systems were not back online, causing delays for FCC customers in filing reports and other documents.

Instead of the IT problems just becoming another example of government and contractor ineptitude, an email obtained by Federal News Radio sent to FCC employees and partners shows the effort to rescue the project.

David Bray
“We could have always asked for more time up front, possibly padded our schedules; instead we chose to be ambitious in our timelines because that’s what a startup-mentality culture does,” wrote David Bray, the FCC chief information officer in the email.

“We aimed high, adjusted, pivoted, and succeeded in our outcomes. If those in public service take from what we did as a ‘Team’ this summer — and from it and see that it is okay to take risks as long as you are fully committed to seeing them through and getting it done — then we will have helped ‘hack the bureaucracy’ for the better.”

Digging a bit deeper into the efforts, Federal News Radio has learned the FCC and contractor team, led by IBM, worked for 50 hours straight to straighten out the problems.

Sources said the problems stemmed from contractors at the IBM data center had mixed up hundreds of server cables creating a configuration that was unworkable.

Instead of blaming the contractor or blaming the government, FCC team members worked 24/7, and in some cases, sources say, slept in the server room.

By Sept. 10, systems were back online and available to FCC customers.

No comments:

Post a Comment