So, I caught a bunch of crap from my coworkers for the wussyness of my post about leaving my current employer to return to Housing. I’m going to give in to peer pressure and write some comments on the projects for my just under 2 year stint at CITES.

Campus DNS
I was initially hired to be Service Manager for the campus DNS service. The existing service is based on ISC BIND running on Solaris hosts managed by the CITES Production Systems Management Group. The backend for data management is a collection of scripts that use RCS, vi, rdist/scp and some perl. Lots of hand editing and proper formatting of text files with little error checking. But, I was supposed to stay out of the existing system as much as possible.
My initial task was to kick off a Request for Proposal (RFP) process to identify and select a vendor and/or appliance system to replace those scripts and systems. This meant collecting input on needs and wants both internally to CITES and from campus. This wasn’t a terrible process, but it was somewhat slow to come together. Next was actually writing the RFP document, working with the CITES Business Office and University Purchasing. That process moved about as fast as it could, reducing the responses to the viable options, getting presentations from the vendors and securing evaluation hardware for some limited onsite testing.
Once a vendor and particular option was selected, we had to wait to get a slot on the Board of Trustees agenda, coordinate with Purchasing to get all our ducks in a row and then wait for the meeting 2 months later to be able to purchase the equipment. The BoT approved our purchase, so we moved forward and the equipment was on site in early January 2008.
Parallel to the RFP process, there were multiple events that caused me to need to become more familiar with the existing system to troubleshoot it. We also had to deploy some additional caching nameservers to mitigate an issue we were seeing from Microsoft Vista clients on URHNet. Some of this was fun, most of it was not.
Being specifically told not to get too involved with the running system, but still being expected to support it and to identify and select a replacement did not engender much trust between myself and the others who have to support the service.
In the end, the DNS Refresh Project will be happening, with anycast query servers coming online in August. The appliance will be going in, but it may not be making the originally hoped for timeline of early August 2008.

uiuc to illinois
In case you haven’t heard, the domain name “uiuc.edu” is being phased out in favor of “illinois.edu”. As part of my Campus DNS Service Manager role, I got tapped to serve on the Technical Advisory Committee for the project. The initial announcement of this project raised much ire on campus so Public Affairs decided to involve a cross section of campus technical folks to try to improve the communication and planning. Noble intentions, but the project as a whole is daunting and has the air of an “unfunded mandate”: someone in power decided it will be, damn the consequences and costs just make it happen.
The bulk of work for this change will fall to the individual department IT staff that have to make a multitude of changes across dozens of systems, “death by 1000 papercuts” as they say.

“Other duties as assigned”
A few months after I started, a long time CSO/CCSO/CITES employee announced his upcoming retirement, so the services that he managed needed to be passed on to someone else.

Real Helix Streaming
This service had been on old hardware running on Windows 2000 Server for several years and got exploited some time in Fall 2006. Since I was the guy in the group that had some Windows admin experience, I was the logical choice to help with the reinstall of the service and take it over going forward. The Production Windows System Management group did the initial OS install, then we migrated the data from an old, failing attached SCSI array to local storage on the new server.
Over time, this service has had some shining moments, but the heyday of Real Server is probably in the past, replaced by the likes of youtube, google video and Flash video. Not a bad service to support, but with few, if any, users it’s not something that gets much attention.

Call Manager 3.3
Another service that happened to run on Windows, so I was the logical person to do it. (To be fair, the job listing I applied for did include a couple lines about the possibility of future VoIP responsibilities.) Of course, being a Cisco product, it couldn’t be just regular old vanilla Windows Server 2000, it had to be Cisco Windows that you can only get updates for through CCO with a valid contract…So it was unsupported and unpatched when it got handed to me.
The system had been installed sometime in 2004 to “pilot” Voice over IP within CITES, there had been some political wrangling between the “Service Manager” and higher ups about who could and who could not have VoIP phones, so it was languishing. I stepped in to support the existing users with the intent of retiring it eventually, but mostly waiting for a catastrophic hardware failure to force the issue. This didn’t happen.
Eventually, many users got moved back to traditional POTS/Centrex service, while a few managed to continue on to the Call Manager included with the Audio Conferencing system below.

Campus VLANs/Networks
The retiring staff member was also responsible for a few VLANS, one for high bandwidth video needs and several that were for the planned, but never realized, migration of campus to VoIP. I had a constant side project of trimming back the “hiband” net to a single node so that it could be less annoying to Network Engineering. That goal has been mostly accomplished. The VoIP related nets are gone completely.

VoIP in General
Since I had been handed the keys to the existing kingdom of Voice over IP, I instantly became the go to guy for any project that involved VoIP.

Shoretel
This was a quiet evaluation of a product that was not well suited for our environment. An IT shop of 10 or an office building with a couple hundred lines that don’t move very often, sure. But 20,000+ lines with dozens of moves, adds and changes a day? No thank you.
The phones themselves were wonderful, probably one of the best speakerphone-in-a-handset I’ve ever used, their voicemail was nice, but the web-based administrative interface was horrible. And the backend was all based off a single Windows 2003 Server box. Again, not bad for a small office building, but do you really want to put more than 5,000 voicemail boxes on a Windows Server? (Other than gigantic Exchange deployments, of course.)
Nice eval, nice vendor, not at all suited to our needs.

Cisco Unified Communications Manager 5.1/MeetingPlace
In the spring of 2007, I got asked some strange questions about what certain pieces of Cisco voice gateway hardware could do. I really had no idea, but I did some research and came up with ballpark answers to the questions. Later that week, it was suggested that I sit in on a meeting with our Cisco and AT&T sales folks. At that meeting, we were handed an itemized list of hardware that we needed to buy to build the joint CITES/ACES Cisco MeetingPlace audio and video conferencing system. Another group of people had been involved in evaluating the purchase: CITES Directors and technical folks, College of ACES Directors and technical folks, Cisco sales, Cisco pre-sales engineers, etc, etc.
This project turned into the “vendor promised the moon but delivered little” hell that everyone dreads. A full year has passed since the hardware was delivered and still no production system.

Agito
Another evaluation for a product we may or may not actually need, they make a device to deliver VoIP over 802.11a/b/g wireless access point and do “seamless” handoff to cellular when you leave WiFi covered areas. Sounds like a better option than installing expensive cellular repeaters (aka “leaky coax”) or microcells all over buildings with poor cellular coverage. But, it only works with a few phones, requires Cisco Call Manager handling all the calls, etc, etc. Not something I was closely involved with, but another headache non-the-less.

I’m going to end this now, no point in lingering on the past, time to look forward to the future.