Fred Johnson, Chingder Yeh, Bruce Ernst

Fred, Chingder and Bruce were nominated as a cross-functional team. They are the storage and database “gods” who recovered the university’s production data without data loss during the Great Gas Blast of 2015. 

They put in many hours into recovering hardware and databases, thus saving the university many hours, days, or weeks’ worth of work to reprocess or manually re-enter data that could have been forever lost. 

Their actions were basically a live Disaster Recovery operation without notice or planning.  They acted professionally, positively, and worked tirelessly, setting a great example for ICT and NMSU.

Congratulations Fred, Chingder and Bruce!

 

Asprey Award Nomination

I’d like to nominate Chingder Yeh, Fred Johnson and Bruce Ernst for this quarter’s Asprey award.

At 8:00AM on Tuesday April 7,2015 the fire suppression system in ICT’s central machine room was accidentally triggered.
The fire suppression system instantly shuts off all the electricity and floods the machine room with inergen gas to snuff out any fires that might exist.

During the “Gas Blast of 2015,” we learned a lot about sound waves, pressure waves, and how to recover when you lose 25 hard drives at once. (16 of them were on the controller primarily responsible for storing our production Banner data)

The great majority of ICT services were available again within 4 hours after we were allowed to start working. Banner took a little longer (12 hours) and banner ODS suffered a hardware failure we couldn’t repair until the next day. (Needed parts)
Chingder and Fred worked tirelessly to bring NMSU back to life. Each of them put in 80 hours of work that week. (And Monday was a normal day)

There is a big difference (to ICT) between being “back” and being “recovered”. (The customer, NMSU at large, sees little difference) So, even though services were available again within a fairly short time frame, there was a iot of other work to do.

I like Chingder’s analogy the best. Chingder said that the first stage (bringing services back up) was like getting everyone to safety after a major flood/earthquake/tornado. Everyone’s accounted for, everyone has a place to sleep. …but, nobody’s back at home yet.

The amount of work it took to rebuild homes, to restructure our Storage Area Network, identify and replace bad drives & data was immense.
Once the servers were alive and behaving themselves, the DBA’s work began. This is where Bruce Ernst comes in to the picture, and the reason I’d like to add him to the nomination.

Bruce took the Banner data that Fred & Chingder were able to bring back to life and turned it into a fully operational, production Banner database. I know Bruce worked many, many extra hours as well.

Bruce, Chingder, and Fred’s professionalism, attention to detail, and passion for their work made ICT look good – and more importantly, provided a platform for the rest of NMSU to do its normal business.

In my opinion, that deserves an Asprey Award.
– C u r t i s