Log in | No account? Register!

Update Hustle and Bustle

Written by CipSoft on Friday 03 July 2009
Every update is always a big event, both for the Tibian community and for CipSoft. Of course, the whole preparation and making of an update takes months but the closer the release day gets, the more hustling and bustling there is. Especially the final week before the release is packed with things that still need to be done, difficulties that still need to get solved, bugs that still need to be fixed and nerves that need to be soothed since everything seems to happen all at once. Shouts like "Oh nooooooo! Arrrrgh!" or "Yippeeh!" can be heard quite frequently behind the doors during these days and the queue at the coffee machine gets longer.

So are you up for a little pre-update tour through the CipSoft headquarters? In the final days before the release, we community managers popped over to the different departments to experience the update frenzy firsthand. We grilled them a bit about it and brought back memories of things that went wrong in previous updates... which made some of them break out in a cold sweat again. Here is our story:



We decided that Akananto's pirate ship, err, office room would be a good start for our tour. As soon as we opened the door, we heard the sound of frantic mouse clicks. Akananto, Craban and our new tester Andrej Delany seemed to be busily engaged in testing the durability of a mouse. Suddenly, Craban and Andrej burst out in celebration and raised their arms just to let them fall again when Akananto groaned in reply that he still cannot exactly say what happened because what must have happened actually cannot happen. At first they hardly took notice of us but when being asked what the fuss is all about they almost drowned us in a flood of information about their efforts to reproduce a weird client crash that is somehow connected to the levers of the elevator in NPC Telas's house. Apparently, it was not the first time they succeeded in making the client crash but since they could not yet nail down the bug's exact location: it was probably not the last time. Since they were about to start the massclicking on levers again, we decided to come back later and headed straight towards the content team's dungeon.

While Knightmare was watching a few testers who were about to complete a quest, we found Chayenne and Denson Larika sitting side by side in front of a screen. They were staring at it in wide-eyed open-mouthed shock since some clever testers just showed them how one of their brilliant ideas could be abused for something evil. However, it is way better to get shocked on the test server before the update so there still is time to fix a nasty bug. Bugs that cause some sort of "unfairness" in the game are very critical. Just thinking about such bugs made us all sweat. Chayenne recalled that it was possible to receive unlimited amounts of cash after the summer update in 2006. Players could get rich by turning in sniper gloves for gold since the NPC did not take and delete those after handing out the money.



So the test server days had been quite stressful for the content team up to then and their multi-tasking skills probably advanced several levels. Besides reading huge numbers of bug reports, bug tracking and fixing, keeping an eye on the change logs, patch logs and threads on the testing boards, they also tried to keep everyone informed of what they were doing, both their colleagues as well as the external testers. To help them recover from their state of paralysed shock, we asked about some more or less "funny" things that went wrong in previous updates and Chayenne told us some good stories.

Several years ago, something went wrong with the items and their descriptions. They somehow got mixed up so when you looked at an item it displayed something different than it actually should have. For example, the dragon scale mail said something like "you hear an evil whispering from inside" and every dead player character had the description "you see a dead troll". The content team received bug reports about that in the style of "ha ha, very funny Stick out tongue" since players thought it might have been a joke! Luckily, it was a rather harmless bug so Chayenne had some good laughs while reading all those bug reports.

She also told us about a less harmless yet funny bug that once occurred with water trickles and poison trickles. Those trickles were designed to be placed over water, thus they swallowed items just like water does with stuff being thrown on it. So far so good. However, someone - who was not aware of that - thought they would make a splendid decoration also over normal, walkable floor. And so disaster took its course. First, the trickles of evil swallowed harmless monsters that happened to pass over them, and actually their barks were still heard although the monsters could not be seen anymore! At least that was what the content team reconstructed from confused bug reports stating that "monsters were yelling from that trickle". Unfortunately, some player characters were lured into that trickle too, in search for those poor yelling monsters... and poof! They vanished from the screen into nowhere. Gulped by a water trickle. But fortunately, we were able to restore those brave adventurers.

Not having heard and witnessed such things before, Ogrey was so intrigued by Chayenne's stories that he decided to stay and listen a little longer while it was time for the rest of us to move on. When we dropped in the office room of the product managers, Craban already sipped his second or third or fourth... well, his nth cup of coffee on that day. On his screen, we recognised several opened tabs and windows: the test board, the email client, the test server with several open channels, ubermega secret lines of world domination code... OK, that last one was maybe a bit exaggerated but it was definitely something we community managers have no clue about.
Craban told us that he had already distributed the bug reports and error logs from the test server among the content team, Akananto and himself that morning. While the client-errors went straight to Akananto to get to know his flaming sabre of bug smiting, Craban kept the server errors for himself. We also learnt that bugs are usually identified quickly. Together with our new tester hero Andrej, he tries to reproduce trickier ones which may take a bit longer though – depending on the nastiness of the little thing they are chasing. So much for that weird scenario we witnessed when we paid Akananto a visit earlier, remember?

Anyway, after the bug identification is done, Craban usually stops by in Stephan's office room to get the server errors fixed - which he had already done that day. He was confident that the bug fixes would be ready for him to upload in the late afternoon. For this, the test servers would have to be shut down for a short time. While waiting for the bugfixes, Craban still had some time left to read feedback from the testers, to take down criticism and suggestions for improvement he later wanted to discuss with various people to think about whether or not something should be changed. In addition, he also answered a few questions on the private test board. At the same time, he followed some PvP testing groups around. Those groups were fighting against each other and shed pixel blood to thoroughly test the skull system. As you can see in his signature, Craban is not a man of too many faces but he is definitely a man with thousand arms, ears and eyes.



Suddenly the door opened, Guido had arrived in the office and was of course quite interested in how things were going. Being the supervising product manager, he keeps a close eye on everything related to the update. Since we did not want to disturb their briefing we tip-toed out of the room and decided to stop by the tech team's room which is close to the product management. Peeking through their door, everything seemed as usual. They obviously conducted their daily business. The team leader of the system administrators explained that they are in alert, ready for action in case of emergency. And they already made sure that there are enough resources to handle the usual update rush.


It was about time to return to Akananto. The king of all test server pirates was still duelling with pesky client errors that occurred on the test server. When we asked him about things that are still on his to-do list he mumbled the magical words "Tibia.exe". Yes, Akananto takes care of bringing all the neat stuff created by the content team into the game by tying two parcels full of data with which you can install the new client on your computers, one for Windows, one for Linux. Of course, he has to make sure that there are no parcel bugs, harrharr. Finally, he has to check if the virus scanner and the new client program get along well with each other or if false positives are triggered. So in case your antivirus software should claim that the official Tibia client is an evil dark lord sent from the abyss to torture your system, do not blame it on the boogie.

We started to quiz Andrej a bit about the last days. He joined our team just recently. Starting to work as a tester right before an update really is a challenge but he is already doing a great job. Of course, his main tasks before the update involve all kind of things related to testing, bug tracking and fixing. After so much technical information we were up for some cool artworks. Andrej continued mulling over strategies to improve the internal test and we rushed towards the graphic kingdom. Of course, we did not forget to pick up Ogrey who was totally enthusiastic about all the great stories the content team just told him.



Jan, Tibia's fabulous graphic artist, was still busy with the new header artwork for the Tibia website when we popped into his room. And guess who was there? Craban and Guido were looking over his shoulder. We heard the three of them exchanging ideas and Jan made them come to life by drawing them on paper. There is always something to draw, and there is even more when an update comes closer. While the new ingame graphics for Tibia have to be ready by the feature freeze several weeks before the update, some adjustments still need to be made during external test phase. In addition, Jan is also responsible for the great teaser pictures and the update collages in the news article.

After having heard and seen so much, we finally set out for our little CM home in the attic but of course not without stopping by in the customer support department. While most of them also conducted their business as usual, Manina seemed to be engrossed in adjusting the documentation on the website: the library, the manual and the FAQ. So we moved on to our office room with a short stopover at the coffee machine since two of us had to refill their engines. Back in our office room, we resumed our daily tasks. Usually, all teasers are already finished and published when the last week before the update starts so our update related tasks mainly focused on writing the update news article and taking care of the teaser feedback threads.

Later that day, we talked a bit about the day of the update. We all work towards that day but it is usually a pretty quiet day for several departments since the whole update stuff has to be completely finished by that time. So most of us mainly focus on other tasks keeping fingers crossed that the update process will go smoothly and as planned. Of course, Akananto is in alert in case Guido or Craban come running to him in panic with a technical problem. The content team is also prepared for really urgent bug fixes.

For the product managers, however, the day of the update is actually 2 days long. A day before the update, Guido and Craban already prepare everything and go through the entire update process once on a special test server on which they copied the backup of one random game world. After the process is completed, they also hop on the test server to check whether everything looks like it is supposed to. If everything is OK, they finally change the settings for the update from target "test server" to target "all real servers". All of us are almost bursting with excitement at that time.



On the day of the update, the product managers come in early to get the files for the website and the new header artwork. They sit together once and quickly go through the plan again. Tension is in the air! At the announced time, the update is started by shutting down the website and the login servers, then the rest. From now on, Craban and Guido basically copy files around and hit start buttons for the different steps of the update. Once the update process is finished, they check online if everything looks fine before they switch the website back on. Finally, backups and the update files are put into an archive and the internal documentation gets updated. Please give a warm welcome to this year's summer update!



And now it is your turn to explore all the new features and the new game content. We wish you a great time and hope you enjoy the update.

Rock on!