Measuring the success of TCoE

Measuring the success of TCoE

Listen on the go!

“What gets measured, gets done”. This could not be truer after setting up a Test Center of Excellence (TCoE). A TCoE is setup to centralize testing resources, tools and practices in a large enterprise and offer testing as a shared service across multiple groups. Enterprises need to constantly prove the value of retaining the TCoE and increase the management’s commitment to further develop the TCoE. In this paper, we out-line the different metrics that measure the business benefits delivered by the TCoE.

Why TCoE?

Traditional “siloed test teams” are distributed across multiple business units (or) groups and they are governed by different managers. Because of this organizational design, test teams cannot ensure testing consistency across the organization as they are controlled by managers who have their own priorities and political realities about the importance and budget allocation for testing. This led to massive inefficiencies across the organization in terms of costs, reusing best practices, quality of testing, effective utilization of teams, hardware & software. A testing centre of excellence solves this problem by the centralization of all test assets (people, hardware infrastructure, software licenses, process assets) across the organization and offering “testing” as a shared service across multiple lines of business units (or) groups. Collapsing multiple test groups into a centralized testing group will lead to better efficiencies due to economies of scale & optimization. Testing Centres of Excellence (TCoE) (Ref Fig 1.2) framework ties all the strategic assets of the testing such as people, process, hardware & tools which are held together through enablers and a robust governance model. Transformation of a “siloed testing organization” to a “TCoE based testing organization”.

Measuring success of TCoE

There is a need for a measurement framework that can compare the intended benefits versus actual benefits to demonstrate the effectiveness of TCoE. It should be comprehensive and cover all aspects like people, costs, quality, test maturity and speed. A TCoE promises the following benefits for an organization.

  • Reduction in testing costs
  • Improved quality of applications in production
  • Improved test maturity
  • Better productivity
  • Improved time to market
  • Improvement in overall skill-set of the team

Based on these six benefits and our experience of working with enterprises, we recommend the following framework to measure the effectiveness of a TCoE (Ref Table 1.1)

  • Testing Cost
    1. Average cost of testing per hour
    2. Resource utilization
  • Quality of Planning
    1. Planning performance
  • Quality of Product
    1. Defect leakage into production
  • Test Maturity
    1. Test maturity rating
    2. Test process standardization index
  • Competency of the team
    1. Skill-set index rating
  • Infrastructure Cost (hardware & software licenses)
    1. Over all infrastructure costs
    2. Utilization

Focus area

Metric name

Purpose

How to calculate

Testing Cost Average cost of testing per hour Costs saved in testing by establishing TCoE Compute the average cost of testing before and after establishing TCoE
Resource utilization To measure the improvement in utilization of resources It is computed by dividing the utilized hours with the total number of hours available
Quality of Planning Planning performance Improvement in forecasting of resources This is measured through effort variance and schedule variance of testing projects
Quality of Product Defect leakage into production Improvement in quality of the applications that are deployed under production Weighted average of number of defects found in production
Test Maturity Test maturity rating Over all maturity of testing practices Computing the overall maturity using TMM (or) TPI (or) internally developed system
Test process standardization index To measure degree of standardization achieved Over all process standardization
Competency of the team Skill-set index rating Improvement in overall competency of the test team Skill-sets required VS available
Infrastructure Cost (hardware & software) Over all infrastructure costs Costs saved in testing infrastructure costs Compute the overall testing infrastructure costs before and after establishing TCoE
Utilization Improvement in utilization of the hardware and software Compute the overall utilization of hardware and software licenses usage before and after TCoE

Conclusion

Many enterprises have established testing centres of excellence to improve the release quality of the applications (or) products in production environment, deliver better business value to the internal customers of testing, improve the overall maturity of testing & optimize the testing costs across the enterprise. Unless the promised benefits are measured against the actual delivered benefits, the CIO/Head of Testing will not be able to demonstrate the effectiveness of the TCoE. This framework can help the IT leadership in an enterprise to measure, showcase and in some cases figure out what needs to be done to really deliver the intended benefits of a testing center of excellence.