Satisfied with the quality of testing Planit was already providing them, the telematics company enquired requested our support in evaluating and improving their testing and quality practices. With a long history and reputation of delivering quality engineering services and leadership, we stepped up to evaluate the testing practices of the telematics company’s delivery teams and implemented a range of improvements throughout the organisation.

We started with a three-week test practice review (TPR) and identified an initial set of 22 recommended improvements. This was followed by a six-month engagement to implement each improvement, including a range of best practices within each delivery team and throughout the lifecycle.

As stated in the problem statement, one of the key challenges was that delivery teams were conducting non-functional testing late in the lifecycle, which introduced significant security, performance, usability and resiliency/availability risks. Finding those defects late in the lifecycle was proving to be a costly exercise that posed a significant business risk.

The lack of non-functional requirements also meant that test plans, strategies and estimates for in-flight projects were insufficient, focusing predominantly on functional testing, and therefore lacking realistic timelines and budgets. In addition, stakeholders were often not informed of the test approach that was being taken for each product, which made it difficult for them to trust that the right testing was taking place.

To resolve this, we introduced quality risk assessments that enabled teams to rapidly identify quality risks and corresponding non-functional requirements much earlier in the lifecycle. By providing training and mentoring on those practices, we enabled the telematics company to fill the gap in their testing, thereby reducing risk for their business and their customers.

This also allowed them to socialise and plan a much more complete approach to testing with their customers earlier in the lifecycle. This was further supported by defining a standard set of non-functional requirements, and likely non-functional testing costs, as a starting point for all future products.

We also introduced lightweight test plan and strategy templates that allowed teams to design a test approach covering both functional and non-functional testing in a short timeframe. We supported this with training and mentoring in test management practices to uplift the oversight and governance of testing across the lifecycle. Estimated cost savings were in the vicinity of $10,000 per year.

We also introduced the use of security testing tools, to enable much earlier security and penetration testing, reducing the number of security vulnerabilities and reducing the overall cost of quality. This also allowed later phases of outsourced penetration testing to focus on more complex security risks, providing greater return-on-investment. We supported this with security testing training for testers, developers, project managers and delivery leads to bolster their ability to secure their systems much earlier in the lifecycle.

Similarly, to support their teams in conducting the earlier performance testing, we provided their delivery teams with training in this type of testing and showed them how they could introduce it much earlier in their lifecycle.

There was also little evidence of previous cycles of disaster/recovery, availability and resiliency testing. We assisted their infrastructure team with developing a test plan for this testing along with high-level test cases, with a recommendation that their team record and report test outcomes to ensure evidence was available for future customers.

We also supported the telematics company with introducing usability testing for their apps, and with developing guidelines for usability testing, which allowed them to identify usability issues much earlier in the lifecycle.

We also supported the telematics company with uplifting their User Acceptance Testing and supported their customers in gaining a better understanding what is expected of them during testing, and vice versa.

The telematics company was also challenged by their existing test management tool, which was integrated with Jira, but could not provide a traceability report identifying which requirements had passed or failed testing across a release. We assisted with developing a business case and pilot that proved that Tricentis qTest would provide with significant cost savings of over $32,000 per year. The tool was subsequently adopted by the organisation.

For in-flight projects, we conducted a test automation review to enhance tests that had been developed in Postman for one set of APIs, and we assisted with the introduction of SoapUI suite for testing another system’s APIs. Recommendations for extending test coverage were offered to each team, with positive improvements being made in the automation approach for each project.

We also noted that testers and developers had a different understanding of unit testing, with neither being fully aware of each other’s viewpoint, nor having a full appreciation of the full suite of early tests could be carried out. To resolve this, we provided training sessions for developers, testers and BAs on black-box and white-box test case design techniques to support more cost-effective testing much earlier in the lifecycle.

A test documentation review was carried out for several projects to provide practical ideas on how to improve and streamline test documentation while still maintaining high quality and continuing to meet ISO 9001 accreditation requirements. We created a variety of lightweight templates to support testing and to provide greater visibility of test plans, strategies, test progress and outcomes.

We also supported the telematics company by facilitating a corporate risk assessment with their executive team to identify a comprehensive set of risks that the organisation could face, allowing them to identify treatment plans for the reduction of each risk.

We also assisted the organisation in understanding what quality means to them by running a survey with staff throughout the organisation, resulting in the creation of colourful “word clouds” that could be showcased on office walls, to remind staff of the importance of testing and quality to them and their customers.

Each improvement was captured on a Testing Services Wiki, including processes, templates, and training materials. This allowed existing testers to utilise the practices for their products, and supported when new testers joined the organisation. Training and mentoring were provided throughout the engagement, to support teams with embedding each improvement into their Agile lifecycle.

We also implemented improvements to several sections of the company’s Business Manual to better support new staff joining the organisation and assist them with maintaining their ISO 9001 accreditation.