Measuring the success of TCoE

Listen on the go!

“What gets measured, gets done”. This could not be truer after setting up a Test Center of Excellence (TCoE). A TCoE is setup to centralize testing resources, tools and practices in a large enterprise and offer testing as a shared service across multiple groups. Enterprises need to constantly prove the value of retaining the TCoE and increase the management’s commitment to further develop the TCoE. In this paper, we out-line the different metrics that measure the business benefits delivered by the TCoE.

Why TCoE?

Traditional “siloed test teams” are distributed across multiple business units (or) groups and they are governed by different managers. Because of this organizational design, test teams cannot ensure testing consistency across the organization as they are controlled by managers who have their own priorities and political realities about the importance and budget allocation for testing. This led to massive inefficiencies across the organization in terms of costs, reusing best practices, quality of testing, effective utilization of teams, hardware & software. A testing centre of excellence solves this problem by the centralization of all test assets (people, hardware infrastructure, software licenses, process assets) across the organization and offering “testing” as a shared service across multiple lines of business units (or) groups. Collapsing multiple test groups into a centralized testing group will lead to better efficiencies due to economies of scale & optimization. Testing Centres of Excellence (TCoE) (Ref Fig 1.2) framework ties all the strategic assets of the testing such as people, process, hardware & tools which are held together through enablers and a robust governance model. Transformation of a “siloed testing organization” to a “TCoE based testing organization”.

Measuring success of TCoE

There is a need for a measurement framework that can compare the intended benefits versus actual benefits to demonstrate the effectiveness of TCoE. It should be comprehensive and cover all aspects like people, costs, quality, test maturity and speed. A TCoE promises the following benefits for an organization.

  • Reduction in testing costs
  • Improved quality of applications in production
  • Improved test maturity
  • Better productivity
  • Improved time to market
  • Improvement in overall skill-set of the team

Based on these six benefits and our experience of working with enterprises, we recommend the following framework to measure the effectiveness of a TCoE (Ref Table 1.1)

  • Testing Cost
    1. Average cost of testing per hour
    2. Resource utilization
  • Quality of Planning
    1. Planning performance
  • Quality of Product
    1. Defect leakage into production
  • Test Maturity
    1. Test maturity rating
    2. Test process standardization index
  • Competency of the team
    1. Skill-set index rating
  • Infrastructure Cost (hardware & software licenses)
    1. Over all infrastructure costs
    2. Utilization

Focus area

Metric name

Purpose

How to calculate

Testing CostAverage cost of testing per hourCosts saved in testing by establishing TCoECompute the average cost of testing before and after establishing TCoE
Resource utilizationTo measure the improvement in utilization of resourcesIt is computed by dividing the utilized hours with the total number of hours available
Quality of PlanningPlanning performanceImprovement in forecasting of resourcesThis is measured through effort variance and schedule variance of testing projects
Quality of ProductDefect leakage into productionImprovement in quality of the applications that are deployed under productionWeighted average of number of defects found in production
Test MaturityTest maturity ratingOver all maturity of testing practicesComputing the overall maturity using TMM (or) TPI (or) internally developed system
Test process standardization indexTo measure degree of standardization achievedOver all process standardization
Competency of the teamSkill-set index ratingImprovement in overall competency of the test teamSkill-sets required VS available
Infrastructure Cost (hardware & software)Over all infrastructure costsCosts saved in testing infrastructure costsCompute the overall testing infrastructure costs before and after establishing TCoE
UtilizationImprovement in utilization of the hardware and softwareCompute the overall utilization of hardware and software licenses usage before and after TCoE

Conclusion

Many enterprises have established testing centres of excellence to improve the release quality of the applications (or) products in production environment, deliver better business value to the internal customers of testing, improve the overall maturity of testing & optimize the testing costs across the enterprise. Unless the promised benefits are measured against the actual delivered benefits, the CIO/Head of Testing will not be able to demonstrate the effectiveness of the TCoE. This framework can help the IT leadership in an enterprise to measure, showcase and in some cases figure out what needs to be done to really deliver the intended benefits of a testing center of excellence.