Rise to the Bell

CHOP-839 - Fix It!

Since 2007, AngusGraham Ceawlin has been a builder of community in Second Life. Over four and a half years, he and his partner Sharrah Brendel have been involved in creating and strengthening historically-focused communities inworld, including World War II roleplayers and a host of pirates and other period seafarers. He has even worked toward layering communities upon communities by founding the AllSeas Battle Group, designed to bring together disparate naval warfare populations. But in December 2011, this lover of community found himself barred from the identity that he’d devoted to so many others.

The Theft of Identity
Angus was uploading textures when he experienced what he probably expected would be an ordinary crash-to-desktop. When he attempted to log back in, however, he lasted only a few seconds before Second Life booted him back out again.

Several more attempts on various viewers later, being a Premium member, he contacted Second Life live chat for support. There were hoops, there was jumping, and there was a whole lot of silence. Somewhere along the line, the ticket was escalated to Linden attention while the JIRA he was told related to his case, CHOP-839, remained inaccessible to the public, including to Angus himself.

On February 3, after he had been unable to access his account for a month and a half and had procured minimal information about the status of the mysterious CHOP-839, Angus created his own JIRA issue, SVC-7653, upon head of support Izzy Linden’s recommendation. All he knew from Izzy was that Second Life support had determined that his problem was caused by an inventory error, but the script that was normally used to fix this kind of error was broken. He was essentially waiting for the inventory repair script itself to be repaired, following which his inventory issue could be fixed as well and he would be able to log in.

What no one at the Lab could tell Angus was how far away this tool actually was from a fix. In the meantime he had no access to his main account’s group ownerships, land rentals, inventory contents, and–most distressingly–identity. After the SVC JIRA was filed, however, the public side of his case ballooned. Friends, acquaintances, and concerned strangers offered words of support in the comments, while the ticket accumulated thirty votes and thirty-two watchers as of Tuesday, February 21. A thread on SLUniverse was begun, and discussion cropped up in Rodvik Linden’s feed, while “CHOP-839” became a minor Twitter meme.

Angus has been active in the pirating/seafaring community for many years, frequenting swashbuckler hotspots like Blake Diego.

Most importantly, it came to the attention of the Phoenix/Firestorm team. Support team member and JIRA guru Whirly Fizzle contacted Angus to investigate the finer details of his predicament and pulled in developer Nicky Dasmijn, one of the team’s most able and avid bug-squashers (if you saw a reduction in memory-related crashes in the most recent Firestorm release, you probably have Nicky to thank, and she developed the viewer’s mesh uploader).By the end of that weekend, on February 19, Angus was logged into a Second Life grid for the first time in two months on a version of Firestorm custom-patched to address his particular issue. It was the beta grid, and he was not able to receive inventory offers without crashing, but it was the first sign of progress Angus had seen since his ordeal began. A day later, after Nicky had made further adjustments, he could not only log in but view his inventory on the beta grid as well.

Angus poses with a chair he rezzed and a texture he uploaded thanks to the custom-built version of Firestorm.

Needless to say, the success had Angus gushing in the JIRA and to anyone who asked–including myself when I spoke with him on Skype–about the effort Nicky and Whirly had put into diagnosing and treating his issue. Nicky left a comment on the JIRA explaining in very comprehensible terms what appeared to be the problem and what would need to be done at the Linden end to allow Angus to log in and function on the main grid. Twenty-four hours later, however, there is still no Linden response on SVC-7653 or indication that anyone at the Lab has taken on Nicky’s suggestion.

A Mudwrestling Match I’d Buy a Ticket For
It may be tempting to read this saga as a triumph of the open source community in the face of Linden inadequacy, but from my standpoint as a member of the Phoenix/Firestorm support team myself, the case looks a bit more complex.Izzy Linden had Angus file SVC-7653 so that the issue might end up on a developer’s radar, as Angus wrote in the original JIRA summary. Although a third party developer was probably not the particular kind that Izzy had in mind, it serendipitously worked out that way. The unintended consequence was that the Phoenix/Firestorm team finished in two days what the Lab gave no indication of even beginning in two months. But why?

I don’t believe it’s a case of “our devs can beat up your devs,” though that’s a mudwrestling match I’d buy a ticket for. Rather, it’s about the kind of support and development assistance each team is structured to provide and the unfortunate tradeoff between formal organization and flexibility.

What worked in Whirly and Nicky’s favor were 1) open lines of communication between support and developer, and 2) the ability to institute the unconventional workaround of creating a custom viewer. Under normal circumstances, support works with users on problems that can be fixed at the user’s end, while developers work on problems that require changes to the viewer itself. When a development fix is not going to be immediately available, support may help users find workarounds to let them function in the meantime. Angus’s is a case in which the division between support and development was not as firm as that between fix and workaround.

There is no “pure” support workaround for a broken account, but Nicky was able to fill that gap by patching a viewer, blurring the line between development and support. The Lab still needs to implement the account-level fix, as third party developers have no access to accounts, but at least Angus can log in and open his inventory on the beta grid and can feel like “himself” again. A flexible organizational structure–one in which members are volunteers who choose what projects they want to take on and who tackle tasks because they sound challenging or they want to help others and not because it’s part of the job–was necessary for this to take place. This is the kind of flexibility an open source, community-oriented organization can easily have; although it probably isn’t impossible for a large business to have such a flexible structure, it certainly does not seem to be the norm for companies in general or the present-day Linden Lab in particular.

Take the blockades that stood between Angus and his account in comparison. He had to communicate with support in order to learn that a developer needs to fix the tool that support, at that point, can use to fix the problem. The shortcomings here don’t necessarily lie in any individual’s (in)ability to perform his or her job but in the excess layers of who needs to do what and when. According to the timeline Angus explained to me, Linden support had diagnosed his problem some time between three days after his crash (when they acknowledged it was an account issue) and three weeks (when he learned that the repair would come when CHOP-839 had been fixed).

The real issue, then, was that despite the problem being identified in a not-totally-unreasonable timeframe, no one with whom Angus was in contact had the ability to work on or encourage work on CHOP-839. What support can do, however, is point the user directly toward dev channels and, in fact, that is what Izzy appears to have done when he instructed Angus to file a JIRA under SVC. Thus, Linden Lab evidently lacks the two attributes that allowed a volunteer-run organization to succeed: the flexible lines of communication and the ability to diverge from a standard range of solutions.

Saint Quentin is a World War II region Angus counts as one of his primary centers of community.

Community, a Redux
I would not presume to say that Linden Lab ought to restructure to become more like a volunteer team. Though they could perhaps pick up some ideas, the Angus story is not strictly about one form of organization being universally better than the other but merely better for this particular circumstance. In fact, what I consider most valuable about the way the situation has turned out so far is understanding how the availability of multiple systems is what has put the problem on its way to being solved.Second Life is, fundamentally, a virtual community. Even beyond the grid, extending onto the JIRA, to Twitter and Plurk, to SLU, extending to Scouts and Lindens, as well as residents, it is a community. There would be no open source viewer-building community if Linden Lab had not opened the code to permit it to exist. Public JIRA issues allow the kind of collaborative work that Nicky and Whirly’s work will hopefully become with respect to the Lab. Ever community-minded, Angus tied his experience back to this concept: “The best thing about SL,” he told me, “is the potential for people to help people and to be community. These are people I don’t know, and they’re coming to the sound of the bell to put out fire. That’s what community is about, when we rise to the bell.”

A gleeful soldier’s homecoming.

11th Hour Addendum

As I was shooting pictures to go along with this story, Angus received word from Izzy Linden that they were able to repair and run the script that he’d been waiting on. He was invited to log into the main grid and did so successfully. I am pleased to have been able to shoot a very excited Angus in front of the ballroom at World War II region Saint Quentin shortly after he arrived there on his main account for the first time in two months, looking very much like a soldier home from battle.

– Lette Ponnier