Notes from 53rd International STC Conference
Las Vegas, Nevada, May 7-10, 2006

Numbers Instead of Words: Developing Meaningful Metrics for Quality
Round-Table Presentation within Progression, "Winning Management: Managing Projects"

Lori Fisher, IBM

Fisher is a "middle manager" in the user technology department with IBM Corporation. She is also the secretary for STC.

Session Description: The presenter discussed how to measure four basic things: (1) the quality of the finished product, (2) the lack of quality of the finished product, (3) the quality of the process, and (4) the likelihood of achieving quality (predictors).

• The challenge: no common definition of quality in technical communication, no common set of quality measurements for our profession.
• The proposal: four things to measure, a metric for each
• Begin with a definition
   
• To measure something, you must know what "it" is
• There are many definitions of quality: pick one and use it consistently
• Crosby's definition works well: Quality = satisfying customer requirements.
• Develop metrics based on your definition of quality
• What to measure:
   
• Quality of the finished product
• Lack of quality in the finished product
• Quality of the process
• Likelihood of achieving quality (predictors)
• Quality of the finished product
   
• Indicator: customer satisfaction.
• Metric: 1 to 5 scale of overall satisfaction with the documentation (in our case, the communication product/service) based on customer survey response
• Implementation tips:
   
• By phone, comment cards, web surveys
• As "why" if dissatisfied
• Asking consistently provides baseline for comparison
• Lack of quality of the finished product
   
• Indicator: Defects.
• Metric: customer-reported errors per title
• Implementation tips:
   
• Calls to product help line about errors in documentation
• Differentiate comments from errors
• Over life of the documentation, or per month, per year
• Quality of the process
   
• Indicator: Review comments.
• Metric: number of comments per 100 pages reviewed
• Implementation tips:
   
• Substantive reviewer comments, not suggestions
• More is better
• Earlier is better
• Watch for trends, compare to norms
• The earlier you can solve problems via fixing the process, the less costly it is. Once the product is in the field, the corrections are more costly; the longer in the field, the more costly.
• Likelihood of achieving quality (predictors)
   
• Indicator: Editing-based improvements
• Metric: % of editing comments implemented.
• Implementation tips:
   
• "In-process metrics" help predict final quality
• Correlation to final quality
• Editing improves quality
• Measure whether you follow through on edits or not – can be as simple as binary (yes, no)
• After you have measured.
   
• Separate measurement of the product/process from measurement of the person
• Analyze the measurements to see how you can improve next time
• Look for root causes in your process and change the process
• Look for trends – what does measurement mean?
• Start somewhere... start small... but start!