1 of 2 pages
Monitoring Isn’t Enough to Answer the Quality Question
Customer Relationship Metrics conducted a research project that provides proof that monitoring scores do not equal the callers’ perception of service. The monitoring form included 17 items, seven of which could be directly compared to the caller evaluations. We examined the monitor and caller evaluations over a five-month period. There was virtually no relationship at all between the caller evaluation of the experience and the monitoring evaluation.
0 Comments on this document
Document Transcript:
Internal Quality Monitoring Isn't Enough to Answer the
Quality Question
By Dr. Jodie Monger - President, Customer Relationship Metrics
We are continuously asked how well the contact center is serving the corporate asset - how
well is service delivered to customers who call to resolve a problem or to ask a question? In
many centers, we must rely on a summary of operational metrics with the assumption that
certain metric levels answers this critical question. We also rely on internal quality monitoring
(IQM) scores to answer the question.
If your IQM is like most, you have to conclude that most customers are extremely satisfied by
the telephonic service experience. Scores naturally migrate to the upper part of the IQM
scoring scale. If you have 100 points available, the majority of your scores are probably 92 or
higher, or even 95 and higher - essentially you use the top 10 points on the scale.
When attempting to answer the service quality question, basing such an important assessment
on IQM when it has the bias mentioned above diminishes the effectiveness of the response. All
other departments within your organization can report on success with numbers that are not
questioned. The contact center needs such a response - one that is accepted as valid unlike the
monitoring results.
Let's review your IQM program and begin the evolution toward providing a better answer.
Who is doing the monitoring? Avoid the fox guarding the chicken coop. What items are
scored? It's best to focus your monitoring form on objective issues related to call control,
providing the correct response, and effective relationship building criteria. Why shouldn't the
IQM form include customer subjective assessments? Guessing at how the customer perceived
the experience is not accurate and contributes to the inflation of the monitoring scores.
The customer is the best one to answer how their experience went. From a scientific standpoint
you should immediately assess the level of service delivered on a particular call. This is
External Quality Monitoring (EQM). While this rating appears to be subjective because it is not
a hard metric such as ASA or a monitoring score related to the effectiveness of the response
from the company's perspective, the customers' perceptions are the reality that we must deal
with in our centers. If your customers are not satisfied all of those metrics are meaningless. But
yet, if you know how the customers perceive the service delivered and you have a good set of
metrics and IQM scores, the answer of how well your center is performing becomes balanced
and valid.
Customer Relationship Metrics conducted a research project that provides proof that IQM
scores do not equal the callers' perception of service. The IQM form included 17 items, seven
of which could be directly compared to the caller evaluations. We examined the IQM and
EQM scores over a five-month period. As presented in the table below, there was virtually no
relationship at all between the caller evaluation of the experience with the EQM program and
the IQM scores. The only statistically significant relationship was related to perceived interest
Page 1
Copyright © 2007 Customer Relationship Metrics, L.C.Monitoring Isn't Enough to Answer the Quality Question
in helping and tone, and this was not a strong relationship.
The results of this research had a dramatic effect on the Quality Assurance Program. The proof
from the customers' perspective that the IQM form was not effective underscored the need to
have a valid answer to how well service was delivered. In addition to a better answer, a
significant savings was now possible.
The original IQM program included 17 items scored per call, 5 per month for 2000 agents. This
equated to 170,000 scores given per month, with 4 completed per hour, taking 2,500 hours (not
including the feedback time). To complete 2,500 hours of scoring, 17 FTE were used at
$45,000 per year for a grand total of $765,000 (again, without feedback and coaching time).
With the results of this research, the IQM form was revamped to focus on objective measures.
Scoring eight items allowed six to be completed per hour, requiring 12 FTEs at $45,000 per
year for a net personnel cost of $540,000. The improvement in the process yielded a savings of
$225,000.
Your own situation may be on a smaller scale, however the relationship of the direct benefit
would apply. Savings from the actual time spent on scoring is compounded by the result of
having a more effective definition of quality. Your three part answer needs to include: 1. Call
Metrics, 2. Internal Quality Monitoring, and 3. External Quality Monitoring (An Immediate
Evaluation by the caller regarding the call combined with Survey Calibration).
Dr. Jodie Monger, is the President of Customer Relationship Metrics (www.Metrics.net)
and a pioneer in customer satisfaction measurement for contact centers. Prior to creating
Metrics, she was the founding Associate Director of Purdue University's Center for
Customer-Driven Quality.
Page 2
Copyright © 2007 Customer Relationship Metrics, L.C.
Quality Question
By Dr. Jodie Monger - President, Customer Relationship Metrics
We are continuously asked how well the contact center is serving the corporate asset - how
well is service delivered to customers who call to resolve a problem or to ask a question? In
many centers, we must rely on a summary of operational metrics with the assumption that
certain metric levels answers this critical question. We also rely on internal quality monitoring
(IQM) scores to answer the question.
If your IQM is like most, you have to conclude that most customers are extremely satisfied by
the telephonic service experience. Scores naturally migrate to the upper part of the IQM
scoring scale. If you have 100 points available, the majority of your scores are probably 92 or
higher, or even 95 and higher - essentially you use the top 10 points on the scale.
When attempting to answer the service quality question, basing such an important assessment
on IQM when it has the bias mentioned above diminishes the effectiveness of the response. All
other departments within your organization can report on success with numbers that are not
questioned. The contact center needs such a response - one that is accepted as valid unlike the
monitoring results.
Let's review your IQM program and begin the evolution toward providing a better answer.
Who is doing the monitoring? Avoid the fox guarding the chicken coop. What items are
scored? It's best to focus your monitoring form on objective issues related to call control,
providing the correct response, and effective relationship building criteria. Why shouldn't the
IQM form include customer subjective assessments? Guessing at how the customer perceived
the experience is not accurate and contributes to the inflation of the monitoring scores.
The customer is the best one to answer how their experience went. From a scientific standpoint
you should immediately assess the level of service delivered on a particular call. This is
External Quality Monitoring (EQM). While this rating appears to be subjective because it is not
a hard metric such as ASA or a monitoring score related to the effectiveness of the response
from the company's perspective, the customers' perceptions are the reality that we must deal
with in our centers. If your customers are not satisfied all of those metrics are meaningless. But
yet, if you know how the customers perceive the service delivered and you have a good set of
metrics and IQM scores, the answer of how well your center is performing becomes balanced
and valid.
Customer Relationship Metrics conducted a research project that provides proof that IQM
scores do not equal the callers' perception of service. The IQM form included 17 items, seven
of which could be directly compared to the caller evaluations. We examined the IQM and
EQM scores over a five-month period. As presented in the table below, there was virtually no
relationship at all between the caller evaluation of the experience with the EQM program and
the IQM scores. The only statistically significant relationship was related to perceived interest
Page 1
Copyright © 2007 Customer Relationship Metrics, L.C.Monitoring Isn't Enough to Answer the Quality Question
in helping and tone, and this was not a strong relationship.
The results of this research had a dramatic effect on the Quality Assurance Program. The proof
from the customers' perspective that the IQM form was not effective underscored the need to
have a valid answer to how well service was delivered. In addition to a better answer, a
significant savings was now possible.
The original IQM program included 17 items scored per call, 5 per month for 2000 agents. This
equated to 170,000 scores given per month, with 4 completed per hour, taking 2,500 hours (not
including the feedback time). To complete 2,500 hours of scoring, 17 FTE were used at
$45,000 per year for a grand total of $765,000 (again, without feedback and coaching time).
With the results of this research, the IQM form was revamped to focus on objective measures.
Scoring eight items allowed six to be completed per hour, requiring 12 FTEs at $45,000 per
year for a net personnel cost of $540,000. The improvement in the process yielded a savings of
$225,000.
Your own situation may be on a smaller scale, however the relationship of the direct benefit
would apply. Savings from the actual time spent on scoring is compounded by the result of
having a more effective definition of quality. Your three part answer needs to include: 1. Call
Metrics, 2. Internal Quality Monitoring, and 3. External Quality Monitoring (An Immediate
Evaluation by the caller regarding the call combined with Survey Calibration).
Dr. Jodie Monger, is the President of Customer Relationship Metrics (www.Metrics.net)
and a pioneer in customer satisfaction measurement for contact centers. Prior to creating
Metrics, she was the founding Associate Director of Purdue University's Center for
Customer-Driven Quality.
Page 2
Copyright © 2007 Customer Relationship Metrics, L.C.
