monitoring your datacenter’s temperature

for quite a long time, we didn’t have any type of environmental monitoring in our datacenter. i wasn’t there when our “new” building was built so i don’t know the reason (oversight, i imagine). several months ago, though, one of my co-workers (who arrives way before the rest of us) came in on a monday morning to find that our datacenter was extremely warm (he put his hand on the back of a server cabinet and quickly removed it, like when you touch a hot stove).

it didn’t take long to convince the $boss that we needed a way to keep an eye on the temperature. an acquaintance of mine at boeing recommended the apc environmental manager units, which were relatively cheap. guess what we now have in our datacenter?

what i really like about ‘em:

• quick, easy setup
• syslog logging
• user-defined thresholds
• e-mail notifications
• snmp!

i really, really like snmp. i snmp enable everything i can, and keep an eye on it all with various tools (zenoss, cacti, homebrew perl scripts, etc.).

we had an incident where the apc unit didn’t alert us like it was supposed to. that led me to spend 15 minutes banging out yet another perl script. this one, ran every 15 minutes from a crontab, polls the temperature oid of the environmental manager. what it does depends on a few things:

• if the unit doesn’t respond, an e-mail alert is generated
• if the unit responds and the temperature is below the threshold “x”, nothing happens
• if the unit responds and the temperature is above “x”, an e-mail to our internal list is sent
• if the unit responds and the temperature is above “x+5″, an e-mail to our internal list is sent, as well as an e-mail to our facilities staff’s internal list

the multiple levels of “escalation” ensure that if, for whatever reason, no one in my department responds, our facilities staff can be alerted and hopefully one of them will address the situation. the nice (arguably) thing is that as long as the temperature is above the threshold, the e-mails will continue to go out every 15 minutes until someone takes action. it doesn’t take long before those e-mails get annoying — especially for those of us who carry a blackberry 24/7!