blogging about quality assurance

We are getting ready for E3! There should be some great games on display, to keep up with all of the blockbuster movies coming out this summer. Avengers, Dark Knight, Expendables… So much good stuff!

The video game industry is undergoing some challenging transitions (that’s about the nicest turn of phrase we can muster).  QA testing groups must adapt to the transitions as well.  We are seeing smaller testing teams for longer periods of time on the games we’re currently testing.  This is great in a way, because our testers get very familiar with the dev team and the project.  When our client is pushing new updates of a game every week or two, we can continually refine and improve our processes.

But after a while, even the most die hard tester needs a little change of scenery.  We like to keep them happy, because a happy tester is a… well, a happy tester.  But we have a few big, beefy men who have been testing ”Rainbow Butterfly Hair Adventures” for a few months.  Months.  They would give their left arm to work on a first person shooter.  But that would make the control scheme a little tricky.  We generally frown on that.

So, we prefer to rotate our testers from time to time to get fresh eyes on a game.  This ensures that the same old, same old way of testing a game won’t bypass a nasty bug.  Plus, we like to spread the wealth of knowledge and experience around.  Some handy time-saving technique that works on Rainbow Butterfly might benefit another title.  If everyone has their noses buried in one game, every day, that experience is never shared.

Of course, we don’t drop fresh testers onto established games without support and training.  We typically have them “shadow” the established team, and then shift onto the project during a lull.  In this business, there aren’t a whole lot of lulls, so we have to be nimble, but it can be done and the benefits can be substantial.

The producers are usually the hardest people to convince about this technique.  I’m a producer, and I know I always want my trained, trusted testers, forever.  But it’s important to understand that fresh testers can help a product succeed.  And besides, those trained, trusted big beefy guys aren’t going anywhere.  They just slid their chair about 48 inches to the left of where they were yesterday.  If you need them, they’ll be there.

As reported previously, we’ve expanded our video game testing service to include cell phone games… iPhones, iPads, and Android apps.  We love these projects, in part because they don’t require huge amounts of expensive hardware.

But where do you get the hardware?  It’s very easy when your game is a PlayStation 3 game.  You just take a few wheelbarrows full of gold to Sony, and you get a test kit, same with Microsoft or Nintendo.

But what about an Android device?  Who is the manufacturer?  What is the official hardware spec?  We have phones from:

Motorola
Samsung
LG
Kyocera
HTC
Acer
Sony Ericsson
Huawei

And that’s not a complete list of manufacturers.  There are all kinds of oddball manufacturers out there, as well as a few not-so-oddballs like Dell, Garmin, and so on.  Each manufacturer has several devices… There are 24 for Motorola alone.

And each kernel version is a little different, and each browser is a little different.

"Dude. You need to upgrade to 4.0 Ice Cream Sandwich."

What is an earnest and thorough QA group to do?  Well, you need to get an assortment of phones, manufacturers, and operating systems.

You do get some new phones, and any day that you get to go shopping for a smart phone is a fun day. Nothing quite like tax deductible retail therapy! But you don’t want to limit your testing to brand new phones. You need to buy up a bunch of used phones, so that you are testing on what most people out there are using. So where do you go?

Craig’s List!

I confess to having an unhealthy obsession with Craig’s List.  There are a lot of crazy people out there with crazy things to sell or give away, and I have to read about them into the wee hours of the night.  I am intrigued. Just what is in that “Curb alert!” or “Garage sale leftovers!”  Where can I dump dirt for free… What horrid grammar and spelling mistakes will be found in the listings?  ”Swing has broke but I think it could be fix.”

Back on topic.

If you’re buying items on Craig’s list, use caution.  There are about a bazillion people who buy things from people on Craig’s list with no problems at all.  There have been precisely zero newspaper stories about me and the nice dining room table and chairs that I sold to a nice young couple a few years back.  But there have been a few evildoers who get a lot of publicity. Be smart and safe.

Before you go off to someone’s evil lair to buy their old phone, a few tips:

If you meet someone at their home, you are less likely to be stood up, and less likely to be buying stolen property.  Just be sure someone knows where you are going and when you’ll be back.  Even better, go with a co-worker or friend who can wait in the car.  And go during the day.  I’ve also arranged to meet people at Starbucks, which is a good second choice.  It’s a very public place, and they are everywhere.  Not crazy about the coffee, but that’s just me.

Check model numbers before you go.  Most manufacturers have some standard naming conventions, so you might save yourself a trip by getting the numbers first.  Apple was quite sneaky with the iPod Touch.  There was the original, then the second, third and now the fourth generation.  Many people I contacted on Craig’s List thought they had a third generation, but by checking the model number, I found many that were second generation.  It was a wonderful Sherlock Holmes moment to savor, but I actually needed the third generation.  So I had to go right back to my computer to find another Craig’s list seller.

Know competitive prices.  I hate to haggle.  I settle on the price on the phone or by email.  Then when you meet the seller, you just hand over the money and get the phone.

Get original packaging if possible.  This reduces the likelihood that you are buying a stolen phone.  You also get more useless ear buds to add to your collection.

Get a receipt.  Remember the tax deductible retail therapy?  The IRS gets quite irate if you claim expenses without receipts.  My accountant gets even more irate.  So, I have printed out the email communications I had with Craig’s list sellers where price is discussed, and I hope the IRS will accept that as a receipt if we get audited.

The ultimate best situation is, buying a phone from the mother of a spoiled teenager who had an iPod Touch, but just upgraded to an iPhone, and mom just wants some of her money back.  Everyone wins.

A safer, more reliable source of used cell phones are used phone retail stores.  You will pay more, but you’ll get a bona fide receipt for tax purposes and you’ll at least have some theoretical recourse if the phone you buy is busted.  They often give a limited warranty.  You can also get a bunch of phones at once, and save some driving-around time.  Try to find one of these places near a college or university in a nicer part of town, so you have more inventory to choose from.

Enough already with the useful information.  There are some new listings of “free empty egg cartons” and “dead vegetation (palm fronds)” that I need to read more about!

Videogame testers often get sent “on location” to test games at a developer’s office.  It’s an expensive undertaking, and it usually occurs only when a game has a very aggressive release schedule.  In these cases, the cost of sending testers on location can be justified by the time that will be saved.

Tight feedback loop. Imagine this scenario:  A tester working at the QA office reports a crash bug that occurs after you beat the last boss in an 8 hour game.  He carefully documents the fifteen steps needed to recreate this bug with 100% accuracy.  Across the country at the developer, the programmer has this bug assigned to him.  He reads the bug summary, but he is of the Twitter generation, so any string of text longer than 140 characters cannot be understood.  Without carefully reading the steps, the programmer takes a quick peek at the game and tries to recreate the bug on his dev kit, but he leaves out a step, so the bug does not occur.  He shrugs and says, “Not happening on my machine.”  He opens up the bug database and marks the bug “fixed.”

When the next version is posted, the testers read that the bug is fixed.  They get very excited, thinking a code change has occurred.  Since the game is close to completion, the testers don’t have cheat codes.  They have to play through the whole game just the way the end user does.  So, they dutifully go through all eight hours of playing the game and every one of those 15 steps to duplicate the bug.  And what do you know, it’s still there!  So they get angry and assign it back to the programmer.  The programmer gets angry, and again says that the bug is fixed.  And on and on and on.

To be fair, it is quite possible that once or twice in the history of quality assurance testing, a tester might have missed a step when writing up a bug.  Or possibly a tester didn’t write the steps with laser precision that we train our testers to use.  I will grant that this is a possibly possible possibility.  However, in the unlikely event that this occurs, the end result is the same.  The programmer can’t reproduce the bug, and if he or she can’t reproduce it, it won’t get fixed.

But wait! If the tester is in the next room, he can drag the programmer to his desk and point at his screen and say, “Look.  Here is the bug.”  Or better yet, the tester can sit at the programmer’s desk and recreate the bug on the dev kit, so the debugger can catch it.

And when the tester gets this chance to demonstrate the bug, there it is… as plain as day, that ugly crash bug.  Oh.  I. See.  A blanket of silence falls over the room.  The programmer hangs his head in shame and slinks off to flog himself (but not until he changes that semicolon to a comma).  Takes him all of 15 seconds.  Multiply this by your bug count, and you can see how much time you can save by having QA testers work in the same office as the development team.

Working at a developer is a great way to learn about how games are actually made.  You’ll see how the game development engine works, how builds are pulled together, how textures are changed and geometry of the environment gets tweaked.  You’ll see wire frames of the game levels, collision boxes around characters, and data tables and animation engines and how to adjust camera… It’s like going to game development school without the student loans.

And the best benefit of working on location… you can expand your network of contacts, so in the future, when the developer is staffing up, and they need a junior level designer… they might just call that reliable, professional, smart and capable tester.  Or they might refer you to someone who is hiring.  You never know, so you’d better connect on LinkedIn.

More good things about going on location… if you are sent off site to test, that means you are critical to the team, respected by The Boss and deemed responsible enough to handle the pressure and stress away from home.  And there will be stress. You are isolated, away from your family and friends.  You tend to just work and sleep.  When you do get a break, you tend to go a little bonkers, to relieve the stress.  I’ve seen it with my own eyes.  The first tester I went on location with ended up cleaning his barf out of our rental car after a night of “stress relief.”  Yuck.

Be Prepared.  Bring a laptop for email, because you might not get to use the developer computers for your personal email.  Bring some books, some movies, things to entertain you during off hours.  And pack some clothes.  Seems a little obvious, but you never know.  I went to Seattle with a tester once, and he didn’t pack any clothes.  He just had the clothes on his back.  I asked him, “You do realize that we are scheduled to be here for two weeks, don’t you?”  And he said he did, but he just didn’t think he needed to bring clothes.  To this day I don’t understand what he was expecting.  Was he just going to wear the same shirt for 14 days straight? So, instead of going to the developer to report to work, we had to leave the airport and go to the mall to buy him some clothes.  And then he got all picky, because the pants weren’t really quite right, not quite baggy enough. That’s when Nice Producer disappeared and Mean Producer came out and barked at him to shut the heck up and just buy some dang clothes.

Speaking of clothes, if you go test at a developer, be prepared to loosen your belt a little. The most frequent problem with testers on location is weight gain!  One of the perqs of working at a developer can be free sodas and frozen burritos.  Seems like a perq, but it’s really a just a clever way to keep you in the office a little longer.  They usually offer this to the test team, too.  The testers can get a little delirious about this, because we at DDF are notoriously frugal about everything.  Stop breathing so much, people! Oxygen ain’t free you know!  Wait.  What?  It is free?  Hmm.   Well, leave some for the rest of us!

And the hotels?  Oh, man.  They are losing money on us.  Our hotel has free breakfast.  And not some lame continental breakfast with bad pastries and a bowl of apples, but a real, full-on breakfast. Once our team checks out and the hotel gets wind of the breakfast bill, they might want to review this policy.  Take a look at this order of bacon… for one guy.

Just a typical light meal before a day of work.

So, yeah.  Back to the testing.  It’s great to work so closely with developers.  We love it, it’s exciting, and it’s a great opportunity for our testers to branch out a bit.  But we miss our guys (and gal).  Be good, be safe, and come back soon!

The thermometers are hovering around 108 just outside our delightfully air conditioned offices.  It’s back-to-school time in Arizona, and families are coming back from vacation, buying spiral notebooks and color-coded folders and getting ready for the normalcy of the school year.

But in QA, we are all cranked up for the big Christmas crunch time.

Most console video game sales occur during the Christmas shopping season, which is supposed to start the day after Thanksgiving (although it sure seems like the Fourth of July stuff is just getting put away, when you know who comes right down Santa Claus Lane). Retail decorating traditions aside, unless your game is downloadable, it needs to be done in the summer to make Christmas.  In this latest post, I will try to explain why.

When is a game really “done” anyway? I don’t know of a game where the developer didn’t want to add just one more feature, or fix just one more bug, or tweak the controls just a little more.  But, life must go on and more importantly, the budget runs out.  At some point, the games are proclaimed “done enough,” usually by an exhausted producer like me.

Then the games need to get approved.  You know at the beginning of a game, there is a long parade of logos?  Comic book publishers, game developers, publishers, movie studios, TV networks, software licenses, etc etc etc? There is at least one person, sometimes several people, at each of those companies who needs to review and approve the game before it is deemed “done enough.”  And that can take a while.  If your logo parade is a long one, allow a few weeks for the approvals.

Once the logo parade has approved your game, the first parties (Sony, Microsoft and Nintendo) take a crack at it.  They run their own QA to test for compliance, also known as Standards.  Understandably, the first parties want to ensure any game shipping on their platform is of high quality and conforms to their specifications.  They make sure you’re not putting in objectionable content.  They check to make sure you didn’t type “NINETENDO” by mistake.  They also want the players of every game on their console to see the same exact text message on screen if someone pulls out the controller while the console has been turned upside down after the game has been in an 873 hour soak test, while muted, during the credits loop.

Maybe I’m exaggerating.

Let’s just say they perform additional testing.  Sometimes games sail through on first submission.  But the first parties often reject games, for all kinds of reasons.  And if they reject your game, you have to fix stuff to make them happy.  It’s their console, so they have quite a bit of leverage over you.  You can imagine, the submission/approval process can cause lengthy delays, so to be prudent, you ought to plan for it to last several weeks.

Then on the frabjous day that your game does get approved, it goes to manufacturing.  This process is controlled by Sony, Nintendo and Microsoft. It’s out of the hands of the publishers, so they generally allow a few weeks for the game disks to get pressed and popped into their little jewel cases, with fancy user manuals and all of those dang cross-promotional inserts.

Then, maybe tack on a week to ship the games in great big boxes to retailers.  And those great big boxes get unpacked, and retailers get the games onto the shelves when they have about 10 bazillion other items to stock.  You don’t want your boxes to arrive on the day before Thanksgiving. They won’t get unpacked for a while.

So what are are we up to… around 10 weeks?  If you want your game to be deep in the retail channel on Black Friday, it had better be “done enough” in August.

And THAT is why QA works so hard during the summer.