Notes from 50th International STC Conference
Dallas, Texas, May 18-21, 2003

Using Metrics to Tell Your Story

Deirdre A. Murr
Walt Disney Imagineering

Deirdre A. Murr, with Walt Disney Imagineering, manages the group that produces the maintenance manuals for all the park's attractions.

Session Description: This session explained how understanding and using metrics gives a manager valuable information on the performance of products, projects, and people. It helped attendees identify which metrics can help tell the right story to their management.

• Why do we need to measure?
   
• What is the quality (value-add) of our product?
• How effective is our product?
• What is the productivity of our group?
• How well is our process working?
• Are we within budget?
• Are we on schedule?
• What is the cost per unit?
• Tell the story and transfer the vision.
• Some common terms
   
• Metric
• Value
• ROI: return on investment
• CRM: customer relationship management (online database)
• Six Sigma: been around 20 years, but really hot now
• Balanced scorecard: finance, customers, internal process, learning and growth
• Definition of metric: A standard of measurement. Followed by Webster's definition.
• Definition of value
   
• A fair return or equivalent in goods, services, or money for something exchanged
• Monetary worth of something
• Relative worth, utility, or importance of something
• A numerical quantity that is assigned or is determined by calculation or measurement
• Definition of balanced scorecard: A management process for measuring strategy; assesses financial, operational, and "intangible" metrics
• Four key parameters
   
• Customers
• Internal process
• Learning and growth
• Learning what your management wants
   
• Do NOT assume you know; do NOT assume management knows
• Corporate strategy/goals and objectives
• Ask other departments for their metrics
• What does management want to know?
• Where is this information going after it leaves your manager?
• My management wants...
   
• Earned value ratio (EVR): percentage showing spent labor hours against percentage of work completed compared to the total budget (baseline: how much money can I return to the project): at Disney, 3% is a good EVR.
• Delivery dates
• What they didn't want to know (but I do)
   
• Average cost of maintenance manual
• How the manual/information is used
• Quality of information
• Value-add to the product
• Success of internal processes of the department
• What is my story?
   
• Determine the results you want to report (i.e., work backwards)
• Where is your department in the maturity cycle?
• Don't begin by collecting at the data on everything (Helen Sullivan, Nortel)
• Find metrics that are useful
• Where do I get the data?
   
• Start with rough historical data and refine it
• Build alliances with other departments who can provide information
• Perform a gap analysis: determine what metrics are missing and create a plan to fill them in
• Best metrics are derived from corporate strategy
• Customer support (problem tracking)
• Marketing (customer demographics)
• Your own group
• Types of data
   
• Number of pages per writer
• Number of hours per page
• Schedule/budget
   
• Percent complete
• Overtime hours vs regular hours
• Hourly rate for each writer/average for department
• Quality and accuracy
   
• Production errors (typos, grammar, etc.)
• Technical errors
• Product improvement
   
• Number of customer problems/questions
• Usability
• Customer evaluation
• Important note: Don't go hog wild on metrics. If you burden your people with 15,000 different types of complex metrics, you will defeat the purpose by undermining their productivity and their morale.
• General approach
   
• Select corporate goals that you can directly or indirectly support
• Establish organizational goals and objectives
• Determine the outcomes and results you want from these goals
• Choose metrics that tell you when you are successful or when you are tracking to success
• Capture data at regular intervals
• Review data, spot trends, reinforce success, make course corrections as necessary
• What you need before you begin
   
• A small group from your department
• A scenario (or ideas about the story you want to tell)
• An overall strategy (corporate?)
• Corporate data
• Facilitating your group
   
• Two areas of discussion
   
• What will you measure?
• What will you do with your data?
• What you should have when you finish
   
• Specific department strategy
• Metrics plan
• Communications plan
• Continuous improvement process
• Some guidelines
   
• Limit yourself to 9 to 16 metrics
• Use more than one metric per product
• Review the metrics and data once a month with your group
• On the first month, use data that is easily accessible/easy to detect, and which is most likely to show success (this will help keep your staff from being threatened by the metric process)
• Add data for one new metric each month (phase in the program; don't overwhelm staff by asking for all the metrics at once)
• Include "intangibles" such as perceptions of quality
• If you can, get information from/about competitors
• After the format for the information is defined, hand off each metric to an appropriate person in your group
• Reporting results
   
• What is the story you are telling?
• Where is the information going?
• What is the attention span of the receiver?
• Does this person like numbers or graphics?
• Deliver data in a consistent format.