There’s an interesting blog post on the official Second Life blog : An Inside Look at How The Ops Team Collaborates. Many a support person the world over will notice the lack of references to expletives, head desking, people asking silly questions and people not wanting to part with information because it will delay them in their aim of fixing the problem, even though that short delay means the helpdesk can provide information to end users if end users contact them.
The use of IRC is strong at Linden Lab and the blog post makes a very strong point as to why communications such as that are pretty damn important :
Running an incident response in a chat channel is also an incredibly effective way of passively disseminating information to a wide audience. A large number of people can quietly lurk in a chat channel unlike in a physical space. More formal status updates to various parties, like support, are of course sometimes necessary but enabling those parties to follow along in real time gives them context that would not otherwise be conveyed in a terse status report.
As a final bonus, we are able to respond to a problem at peak efficiency regardless of where anyone is at that moment. Issues don’t wait for office hours to crop up. Being a distributed team, this is really our only option, but it rocks that being distributed is an advantage in incident response.
Whereas in smaller operations the chat is more likely to be by telephone or shouted across the office, with people openly denying there’s a problem whilst the helpdesk phones are ringing red hot. Then there’s the pause, the WTF moment, the end of the denial, the pass the parcel finger of blame experience and then eventually the fix. Then you have the post-mortem which involves everyone agreeing to be more organised and in full communication mode if this happens again, and then forgetting that was ever said five minutes later.
Linden Lab seem to be very well organised and the fact that they use chat means the existence of chat logs, the use of which are explained well in the blog post. Linden Lab can look back over the logs to identify what exactly happened, they can even educate new staff by letting them read old logs too.
I’m still not 100% convinced that reading the chat log would give one the impression that they were as well organised as the blog post suggests. Linden Lab’s VP of Operations and Platform Engineering, Landon McDowell (Landon Linden), provides the information on how the ops team collaborate.
This is, to me anyway, an interesting blog post. This shows a human side to the people at Linden Lab, which is often lacking in their communications. I’m not saying that Linden Lab employees aren’t humans, although they probably should employ a few Orcs, but at times you lose the feeling of Linden Lab being human because the communications are so clinical.
Linden Lab have a couple of times attempted to let their blog readers meet the humans. They had the Inside The Lab series where Catherine Linden interviewed Lindens. such as Philip Rosedale :
Featuring CEO and founder Philip Rosedale, the podcast covers the growth of Second Life and how Linden Lab has dealt with the technical challenges of scaling the world, the encroachment of the real world into Second Life, Virtual World industry trends, and the growing competition.
Interesting subject matter but Philip gets quoted a lot so the fact that that series also introduced us to other Lindens was good. Then Linden Lab hired Courtney Linden, who was a superwoman when it came to blogging about Lindens themselves in the ON CALL – Support News section of the blog. That section provided support news as well as interviews with many Lindens including Resi Team, Data Linden, Tescalation Team, Mia Linden, Pathfinder Linden, Colton Linden, Niko Linden, Doc Team, Teeple Linden and Matias Linden. Courtney also got interviewed herself. However this allowed people to meet some of the people behind the scenes, it made them seem real and it’s something that has been lost in the evolution of Linden Lab.
I mean back in 2005 Linden Lab were happy to publish blog posts such as : I bling. She blang. We blung. That post was about a party, a real party (there are pics in the linked blog post) which Ramzi, Wendy, Brent, Cyn, Kona, Wilder and Ben Linden worked hard to make happen. I doubt very much whether a blog post featuring the following would get published these days :
The highlight of the evening, at least for me, was when Bill Gurley walked in with a big shiney gold “$”Cash on his necklace. Hawt. Second Best was Bub’s disco ball testicles.
I mean come on! Linden Lab are a lot more professional these days with their communications and at the same time they seem a lot more clinical. In business terms this is no bad thing, they’ve grown a lot since 2005 but it’s a shame that that some of the fun and friendly aspect has been lost. Which is why I welcome that Ops Team blog post, there are humans at the lab still and they respond with the same sense of panic as those humans outside the lab … well at least that’s what they’d have us believe anyway!