Article

How small ATM test teams can get beyond test case cherry-picking

With limited resources, small QA teams must sometimes settle for conducting test cases based on risk and hope everything else turns out all right.

February 22, 2016 by Suzanne Cluckey — Owner, Suzanne Cluckey Communications

Branch transformation ...

EMV migration ...

Cardless and mobile ATM access ...

Value-added services ...

Omnichannel integration.

Today, just about every financial institution in any developed banking market is either contemplating or implementing one or more of these millennial-age musts.

Every project means more ATM testing. And today, more often than not, test plans must incorporate different, but related, systems in order to ensure that an end-to-end transaction will always flow as expected.

But while the quantity and complexity of ATM test scenarios might expand exponentially, small and medium-size deployers rarely have the luxury of expanding their QA teams to keep up.

As discussed in a pair of live webinars this week sponsored by FIS and hosted by ATM Marketplace, limited testing resources can present a set of less-than-desirable options for achieving an acceptable level of ATM security, reliability and user-friendliness.

"When you've got a constrained team, it's very rarely possible to execute an entire test plan every time you want to," said Colin Fraser, FIS product manager and presenter for the Europe and Americas webinar.

"So every time there's a change, you probably know that there are perhaps hundreds — or thousands — of test cases that you want to execute in order to achieve a low level of risk and a high level of confidence that the software or the systems that you're releasing are in a fit state to be used."

He went on to say that a smaller deployer often has to balance available resources against the scope of the change to be implemented and prioritize testing from there.

"For a small change maybe you only execute 10 percent of your total potential test cases and that is driven by risk and your ability to measure that risk," Fraser said. "However it's very difficult to gauge how high the risk remains after you've completed all of your test cases or only a portion of your test cases."

And even if risk is well-mitigated, the question remains whether the change has introduced a production problem that will require additional time and resources to correct later.

"Ultimately it does come down to a bit of guesswork, Fraser said. "You can't say with much certainty that you really have completed enough. So the real question there is how much can you afford to test?"

The webinars covered the various alternatives to manual testing — i.e., outsourced, offshore or automated, and the pros and cons of each: Outsourcing or offshoring can certainly speed up the process, but the deployer loses visibility the testing process and must trust that the provider is doing a proper job.

Far preferable is automated testing can produce the best end result —shorter test cycles, more reliable results and reduced risk, said FIS ATM Senior Consultant Chris Milne, webinar presenter for the APAC market.

Additionally, the deployer can realize unexpected benefits with automated testing, including increased agility and improved staff engagement.

The problem is that test automation requires an upfront investment in money and time. Additionally the deployer's existing QA might not have the expertise required by such a project.

For a deployer ready to try automation, Milne said, it might be best to begin small.

"I know we're going for big gains here, but you might want to start out relatively small — especially for a mission-critical project," he said.

After identifying the right project, the deployer needs to build a business case, taking into consideration not only the cost of the project itself, but also the potential cost of failure and damage to reputation if the project goes wrong. This is where a consulting partner can be helpful, Milne said.

"Usually it's not rocket science. But if they've been there and done it already, then their advice can help you avoid heading down the wrong path to begin with," he said. "As another hint, I would suggest to be realistic in what is achievable.

"We always say that it would be very challenging to achieve 100 percent automation. Maybe a more realistic number is to aim for 90 percent automation and then focus on the areas with the most repetition. Those are the best ones to automate first."

In addition to providing extensive information on testing methods and tips for successful testing automation, the presenters introduced ATM TestBench, an FIS solution for deployers who don't require — or want the expense of — a virtualized or multi-user test solution.

TestBench is a single-workstation system that requires minimal IT infrastructure. The system toggles between manual and automated testing, so teams can run manual tests during the day and execute bulk tests on ATMs in the lab overnight.

The solution features automatic analysis and reporting, and an ATM simulator for repeatable negative testing and requires minimal infrastructure, the presenters said.

Get the on-demand webinar replay.

Download a white paper on ATM TestBench.

About Suzanne Cluckey

---

Suzanne’s editorial career has spanned three decades and encompassed all B2B and B2C communications formats. Her award-winning work has appeared in trade and consumer media in the United States and internationally.