So you're on the hunt for test automation services for your web or mobile app, right? But what is it that you need, exactly? Have you defined your requirements or are you willing to be persuaded by the glossiest marketing brochure?
There are thousands of test automation service providers willing to grab every last dollar off you. But there are a few things that you need to be sure about, before signing long term (or even short term) agreements with these service providers that put your professional reputation on the line. In fact, there are 5 key decision points you must consider before you make this decision.
Understand the context
What is it that you're putting on the line with this decision? Is it the quality of applications that allow your customers to spend more money on your goods or services? Is this an application that your team members or vendors use to make sure that you are actually able to service your customers? You must understand the size and ramifications of the opportunity before you can effectively decide on the most suitable test automation services provider.
Most people in your shoes look for external help with test automation because they identify with at least one of these factors:
- A bad release that has cost lots of revenue, profits or reputations;
- Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;
- Upgrades are long delayed because quality can't be tamed; or
- Your customers are switching to the competition because your application has too many bugs in production.
We know this because of two reasons, a) our customers told us about these problems when they started using our test automation services, and b) all of my colleagues in our leadership team have faced these issues in our previous lives. Boy, do we have some embarrassing stories on this front!
You are definitely not alone in facing such issues. You're also a step ahead of the competition because you are doing your own research about the best solution for you, rather than jumping into the arms of the first vendor who tells you what you want to hear.
Your decision should be influenced by 6 key factors that go to the heart of every good relationship between an organisation and its test automation services provider.
Testing tools
Do you have existing testing tools that you are mandated to use? Are you prepared for your testing services provider to recommend and use tools that may be a better fit for the web and mobile applications that need to be tested?
The issue of automated software testing tools is central to any such engagement. Choose the wrong tool and you are effectively building many future inefficiencies and frustrations into the project. By selecting the automated testing tool that is best for your needs you will lay the groundwork for all parties to be more effective in performing their role and you will should also be able to reduce costs during the engagement.
Remember, previously popular tools provide the greatest incentives to service providers, but they haven't kept pace with testing needs of fast-paced digital environments that are designed to meet very exacting customer standards. To test digital web and mobile applications you require functional testing tools and load testing tools that are built for this purpose and will scale with your development needs, without requiring you to blow exorbitant amounts of money just to keep their infrastructure alive and up to date.
Our Ultimate Guide to Test Automation has a comparison of testing tools that you will find helpful. Our Secret Toolbox has best-practice tips that you must insist on using during your engagement.
Here's a comparison of the most popular testing tools from the Ultimate Guide:
Objectivity in decision making
Presumably you want test automation because you're looking to cut testing time. This is very achievable and indeed something that any self-respecting test automation services company should be able to give you. However, the additional benefit of test automation is that it throws up a number of informative and unimpeachable test-related data that you or your service provider should use when making quality and delivery-related decisions.For example, how do you currently decide when a version of your application goes into production? Most testing programs usually require a test or delivery manager to make this decision based on their assessment of quality readiness.
However, test automation activities, when performed with the right test automation tools, enables you to build quality-related analytics into your delivery decision making processes. These anayltics make possible the transformation of an inherently subjective, role-oriented decision to an evidence-based, data-driven decision where one individual is never a bottleneck.
Want to see how our test automation tools could save you weeks every time you test?
Engagement models
Engagements with test automation service providers have not always delivered the value that customers wanted and expected. The reasons for this vary, but usually revolve around misaligned priorities, capability gaps, technology gaps and poor communication by all parties. In fact this could be true for all aspects of IT outsourcing engagements!
Time and materials was a great model for the service provider of eras past. It provided an endless flow of cash and constantly shifting goal posts meant that accountability could never really be demanded by the customer. You should avoid getting stuck in this vicious cycle.
We almost only engage with our customers on fixed-fee and fixed-scope engagements, where everyone has skin in the game and the customer's downside risk is contained while the upside can be maximised. Most capable test automation service providers will also offer fixed-fee engagements. If they don't then you need to seriously think about why this may be the case.
Communication and integrity
Let's face it: even with the right testing tools and a great engagement model, things will go wrong in a project. Your applications could throw up nuances that will stump even the best automation engineers; user journeys might change after feedback from end-users; there may even be some bugs that are so difficult to fix that timelines suffer blowouts.The key to overcoming these obstacles is communication and transparency. To inculcate these principles into our dealings with customers, we don't provide weekly or monthly status reports in PDF or Excel. Instead we use the real-time reporting tools in our Qsome software testing tools to demonstrate and analyse project progress. The fact that this information is shared in real-time and that our test engineers see exactly what our customers see, we are able to build a greater degree of trust with out customers earlier in every project.
However, not everything can be communicated through a dashboard. Sometimes we have to put up our hand and admit to something that we were trying not resulting in the desired outcome. I find that this transparency with the customer encourages greater sharing of ideas and solutions, which in-turn delivers greater value to the customer.
You must assess whether your test automation services provider is willing to be honest about the project's progress, even when the going gets tough. While honest is a good first step, alone, it is not enough. This is why we undertake to change the way we work, build new features into our testing tools and / or improve our framework when responding to failures. This is the difference between a truly successful customer-vendor partnership and one that produces mediocre, or more likely, sub-standard results.
So what do you actually need?
This is the most important question after all, but you will not be able to answer it without first addressing each of the 6 decision points above. There are three key services that any software testing services provider can offer you. The exact service you need right now depends on how you have responded to the decision points above.- Advice or consulting: a root and branch review of your process from development to delivery. This exercise is often useful prior to engaging a test automation services provider because it allows you to better understand the other elements of the puzzle that can feed into making your test automation efforts a success.
- Test automation services: I call this a "build and let us run" approach where the test automation service provider develops the automated regression suite and leaves it to the customer to execute as and when required. There may be some ongoing commitment from the services provider for maintenance of the regression suite, with a very limited scope.
- Security testing services (penetration testing): this is where you elevate your relationship with your software testing services partner. When the partner understands your applications suite and how your teams like to work, they are in a good spot to also ensure that your applications are secure and that your users and customers can completely trust your applications.
