a couple of months ago i decided to break down and take the cisco ccna certification exams, which i had no problems with. i passed the icnd1 exam and then passed the icnd2 exam a week later.

just prior to the end of our fiscal year (june 30th), we have our annual performance evaluations, which precede any raises we might receive on july 1st. a few days ago, i got my “letter” letting me know what my raise would be (we generally get 3%) and i was happy to see a nice bonus on there on top of that for completing the ccna. if i had known that i would be getting that, i’d have taken the tests a long time ago!

i have to say that the ccna was well worth the (time) investment i put into it, since it didn’t cost me anything financially ($work reimburses me for certification exams). by the way, i never got a bonus for the microsoft certs. :P

for quite a long time, we didn’t have any type of environmental monitoring in our datacenter. i wasn’t there when our “new” building was built so i don’t know the reason (oversight, i imagine). several months ago, though, one of my co-workers (who arrives way before the rest of us) came in on a monday morning to find that our datacenter was extremely warm (he put his hand on the back of a server cabinet and quickly removed it, like when you touch a hot stove).

it didn’t take long to convince the $boss that we needed a way to keep an eye on the temperature. an acquaintance of mine at boeing recommended the apc environmental manager units, which were relatively cheap. guess what we now have in our datacenter?

what i really like about ‘em:

• quick, easy setup
• syslog logging
• user-defined thresholds
• e-mail notifications
• snmp!

i really, really like snmp. i snmp enable everything i can, and keep an eye on it all with various tools (zenoss, cacti, homebrew perl scripts, etc.).

we had an incident where the apc unit didn’t alert us like it was supposed to. that led me to spend 15 minutes banging out yet another perl script. this one, ran every 15 minutes from a crontab, polls the temperature oid of the environmental manager. what it does depends on a few things:

• if the unit doesn’t respond, an e-mail alert is generated
• if the unit responds and the temperature is below the threshold “x”, nothing happens
• if the unit responds and the temperature is above “x”, an e-mail to our internal list is sent
• if the unit responds and the temperature is above “x+5″, an e-mail to our internal list is sent, as well as an e-mail to our facilities staff’s internal list

the multiple levels of “escalation” ensure that if, for whatever reason, no one in my department responds, our facilities staff can be alerted and hopefully one of them will address the situation. the nice (arguably) thing is that as long as the temperature is above the threshold, the e-mails will continue to go out every 15 minutes until someone takes action. it doesn’t take long before those e-mails get annoying — especially for those of us who carry a blackberry 24/7!

i originally wrote the following on 26-sep-2006:

last week i wrote “the joys of a new building“, where i talked about $my_company leasing a building downtown and my job of getting connectivity between there and our main campus.

well, it’s now up and running.

our $vendor had already done their job and tested and verified that they could send a signal both ways. i had one end of the connection set up (the final “patch” from their termination box to our switch), but ended up with an unexpected delay as the other cable didn’t have the right connectors. i ordered the right cable, had it overnighted, and finished up the next day.

i took one of our part-time guys with me to our downtown location to finish up the fiber install and work on some other things. i hooked up the last jumper between the termination box and our switch at that location (an hp procurve 2650, for you geeks) and crossed my fingers.

i plugged my console cable into the switch and my laptop (running linux, of course) and set port #1 to the appropriate vlan for my laptop (my department is isolated on our own subnet and there’s a reservation in dhcp for my laptop). i grabbed the closest patch cable i could find, plugged into port #1 and my laptop, fired up my dhcp client, and watched it obtain the ip address that it should’ve. success!

we’re paying $vendor for the ability to push 100 mbps over this fiber, so that was the next thing i wanted to check. i had previously put an iso image on an internal web server just for this little test. i fired up trusty ol’ wget, feeding it the url of the iso image, and let it go.

in short, 732,336,128 bytes downloaded in 70 seconds. bc tells me that this comes out to 10,461,944 bps, which is pretty close to the theoretical maximum of the link (overhead and such). i’m happy with that!

we moved one user to the building a day or two later and i had him check a few things once his workstation was set up. he reported that everything was working fine. next step is to get our voip phones, configure ‘em, and take ‘em to the site to check things out (qos is set up for the vlan set aside for the voip phones). once that’s done we’ll move the rest of the users to the new site. it promises to be interesting!

i originally wrote the following on 18-sep-2006:

my employer recently signed a four-year lease on a building downtown, about 3.5 miles from our main building. the idea is to move one of our departments (and a few other people) downtown to this new building. the people who will be moving are ones that mostly deal with outside organizations, a great number of which are also located downtown.

my job, being the network guy, was to get a new connection between our main campus and the new place set up. i explored all the usual options: t-1 circuits, wireless, high-speed circuits with a vpn back to the main campus, etc.

just out of curiousity, i called up an isp who has their own fiber running all over — or under, i should say — the city. i told them what i was working on and told them i’d like to know what it’d cost for them to get us hooked up. i told them up front that the cost would probably be prohibitive, but that i was curious.

a few days later, i’m sitting in a conference room with the director of sales, director of facilities, their senior networking guy, and the president of the company. because of my company’s size and the possibilities this might mean for their company (we’re huge), they were interested in doing whatever they could to get us as a customer.

anyway, we meet, discuss my requirements for the project and how they can meet those requirements, bullshit for a few, and we end the meeting. a few days later, i head off to florida on vacation. when i get back, i have a voice mail from the director of sales. i give him a call and he offers to drop by and bring me their proposal.

i meet him in the lobby of our building a bit later and he shows me their offer. i was stunned! they offered to provide us with 100 mbps over dark fiber for $500/mo. — little more than what we’d pay the other vendors for a pair of t-1’s. in addition, they offered to eat the cost of building out fiber the rest of the way to connect the two buildings.

we’d seriously be stupid to pass this opportunity up. i let my boss know what was going on and we went to his boss to let her know. final approval had to come from above, of course, and we got it.

now the fiber is in place and was terminated at both locations late this past week. i hooked up the fiber from their termination box to our switch (at the remote building) on friday and will probably hook up the last fiber jumper tomorrow or tuesday, depending on how busy i am. once that’s done i can start sending bits over the fiber!

i’ve been pretty excited about the whole project since it began a few months ago and it’s neat to see it come to a conclusion. once the connection is up we’ll also be setting up voip phones at the remote location that are connected over the fiber to the pbx at our main campus. voip is new to us, as well, so that should be pretty cool.

i originally wrote the following on 23-dec-2004:

well, i finally got something i’ve been wanting for a while… a dedicated linux server at work.

for a while now, we’ve been running a not-too-critical service on linux and it’s been stable and reliable as hell, but it was running on an old 400 mhz desktop-class pc. finally, it had a hardware failure (i knew it was only a matter of time) and it happened, of course, on the first day of a three-day out-of-town trip for $work. $boss got it straightened out and got the hard drive moved to a new, more modern box, and i was able to tell him over the phone what to do to verify that everything was working properly.

anyway, shortly after that he asked me about a dedicated server, which i definitely wanted. so now we have an hp server running debian gnu/linux in the datacenter. this is a huge step for our campus, since up until now we’ve been primarily a windows-only shop (with the exception of an old novell box or two still lingering around). i like windows and while we never have a problem with the majority of our applications and services i’d love to be able to migrate whatever i can off of windows and onto linux. it’s so much more stable, has fewer problems, and i’m much more familiar with linux administration than i am windows administration.

i’ve been adding a few things to the server already and need to migrate that one $critical_service over, but that won’t be difficult. luckily, the semester is over and there aren’t many students around, since it will result in downtime. i know i hate when a service i’m relying on is down, so i do my best to minimize downtown for everybody else.

anyway, got me a new server. woohoo!

boy, how things change in three and a half short years.

when i started out at this organization, we had that one linux box and that was it. today, we host a number of web sites running apache on red hat enterprise linux. those sites are backed by mysql databases running on gentoo. authentication, authorization, and accounting on our wireless networks is handled by freeradius on red hat enterprise linux. in addition, we have a number of “internal” services and homegrown applications that run on linux servers as well.

more and more of our critical services and applications are running on linux, and windows can’t even come close to matching the reliability and stability of the operating system.

as an example, i have a debian gnu/linux box running syslog-ng that is the syslog server for all of our network devices. it’s uptime just surpassed 500 days. take that, windows!

ever had a really bad day at the office?

these guys have.

jeremy works at a post-secondary educational institution. his job duties include managing high-speed fiber optic networks, administering win2k/win2k3, debian, gentoo, and rhel servers, database administration, security, and occasionally hacking on some perl or php.

he also serves as an adjunct faculty member, teaching network security courses. he is a staunch advocate of foss, is currently working towards a bachelor’s degree in information technology, and also owns a consulting company.

jeremy has been a technological swiss army knife since the oregon trail was text only. his favorite color is sushi. he has never been to the moon.

“the passionate worker doesn’t show up because she’s afraid of getting in trouble, she shows up because it’s a hobby that pays. the passionate worker is busy blogging on vacation… because posting that thought and seeing the feedback it generates is actually more fun than sitting on the beach for another hour. the passionate worker tweaks a site design after dinner because, hey, it’s a lot more fun than watching tv.” –seth godin

(read more)