Notes from 47th International STC Conference
Orlando, Florida, May 21-24, 2000

Measuring the Quality of Technical Publications

JoAnn T. Hackos, Comtech Services, Inc.
Julie A. Bradbury, Cadence Design Systems, Inc.
Constance E. Lamansky, Cadence Design Systems, Inc.

Session Description: This panel discussion on quality metrics is based on the results of the Quality Metrics benchmark study conducted by JoAnn Hackos . Panelists discussed customer satisfaction, measuring quality during development, and measuring organizational and individual productivity.

Quality Metrics - JoAnn Hackos

• Efficiency is doing something well...Peter Drucker
• Effectiveness = doing the right thing well.
• Measuring sales and revenues: making money (contributing to the revenue stream). The ultimate yardstick is the effect on sales: good documentation contributes to the bottom line.
• Customer effectiveness underpins the bottom-line metric. Are we providing our customers with the information they need?
   
• The process
• Measure quality along the way (not just at the end)
• Maintain standards and build consistency
• Increase productivity
• One company had 16,000 calls per month (average 10-15 minutes), five full-time techs, and irate customers. A new customer manual replaced the three (that's right, three!) manuals that were being shipped, none of which did the job. Result was a 50% reduction in calls and much shorter calls.
• Tradeoff exists between depth of documentation and size of tech support staff: sometimes high-level but very clear documentation with tech support answering the unique lower-level-detail questions is the optimum cost-effective blend to meet the customers' needs while maximizing the company's profit.
• ROI = cost to develop new documentation vs. savings in tech support costs
• Process maturity
   
• Level 1: "We don't know why we have problems with quality."
• Level 2: "Must we always have problems with quality?"
• Level 3: Management starts to pay attention to proactive quality control.
• Level 4: Routine attention to quality
• Level 5: Relatively problem-free
• Satisfying customer needs
   
• Baseline customer satisfaction
• Requests for tech support
• Customer access and use of tech info
• Customer access to electronic info sources
• Delivering the right product: identifying information "bugs," measuring effectiveness of technical reviews, validating the accuracy of technical information.
• Without usability testing and verification, there WILL be mistakes...you have missed a key quality gate.
• Somebody who already knows the content will not find the mistakes; customer satisfaction demands customer testing of the product.
• Specific metrics
   
• Competitive reviews
• Competitions
• Treating customers with integrity, courtesy, and respect. Are you compatible with training and customer support; do you have the ability to translate, localize, and customize?
• A well-edited document is easier to translate than a poorly or non-edited document.
• Measuring efficiency: quality assurance (compliance with standards), adherence to process, time on task, cost and value of individual tasks.
• About 30% of total management time should be devoted to planning. For every percentage point below 30% for planning, the project stretches commensurately in time and cost and product quality decreases.
• Measuring efficiency: check accuracy of project estimating, evaluate project complexity, conduct root cause analysis.
• Measuring staff performance and overall productivity of the organization. There is a relationship between employee satisfaction and customer satisfaction.
• Report to management. Metrics-based reporting is much more effective than anecdotal reporting. Relate the total cost of your operation to total sales.

Using Metrics to Manage Information Development: Practical Applications -- Julie A. Bradbury

• What are our best practices? Customer contact metrics. Surveys.
• Objective: Pubs builds expertise on customers and becomes source of information to R&D.
• Train writers on customer contact.
• Use information from customer contact to build a user profile and figure out what products and approaches will best meet their needs.
• Hotline survey metrics reveal which problems are creating problems for the customer and when they will get resolved.
• 80% of customer concerns are involved with content, especially clarity of expression.

Reclaiming Writing Time -- Connie Lamansky

• Customer is looking for improved product effectiveness.
• Two editorial checkpoints were imposed: midway through process and at end.
• Writers do copy edits and minor organizational changes during process.
• Editors get involved with major organizational changes at the beginning of development.
• Minimize meetings and maximize use of e-mail, to increase efficiency.
• Avoid lengthy minutes for meetings; use an action item list.
• Use scripts in the production cycle.
• Eliminate indexes--this elicited cries of protest. A tradeoff study followed to determine when an index is useful and when it isn't...depends on document content, structure, and organization.
• Days saved by activity: Link checks had best ROI, followed by process changes, and elimination of indexes.