The past few days, I've been heads-down, moving one of our infrastructure pieces from supporting a few of our Web applications to also supporting our portal, myEarthlink. This means going from some traffic to quite a bit of traffic. I'd be lying if I said everything went smoothly but (knocking on wood), things do look better and I think we are going to be fine.
I did learn a couple of things, not just technologically but also socially. I'm sure many of these are no-brainers but they were reinforced to me.
Don't rush, Sure everything seems to be blowing up but if you don't take your time you could easily miss a wrong configuration item or something else simple which is causing problems.
Give status as much as possible, when you are working with folks via instant messaging or IRC, it is imperative you let people know what is going on especially if they are waiting on you. The longer you make them wait, the more tension will arrive.
Take care of your own, It's amazing what providing food for system administrators and testing can do to the help being offered.
Know the customer impact, when things are seemingly blowing up, what does the customer see and more importantly sometimes, what doesn't the customer see. This can help you make decisions on when to rollback to previous systems or to just keep going and fix what you can.
Kill your darlings, it's good advice when writing and it's also good advice when working on scaling systems. Many times you have something which was either new or not used much before now. If you hold on to every bit of cool technology at the expense your user's experience, no one is going to be happy. Be able to admit that, get something else working and move on.
Last month, many people talked about the data eBay provided about how they scaled to support the traffic they currently have. In Frank Sommers' post, one thing which jumped out at me was eBay's willingness to change how they were doing things. They moved from a very primitive early system, thru a J2EE architecture to what they have now. You always need to evaluate your systems. Sure, moving from one architecture to another isn't easy and sometimes people have their own agendas in trying to do it but it is your responsibility as a developer to focus on the how this system affects the user and what changes are needed to make things even better for them.
Technorati Tags:
scability, architecture