Outcome:
Complete improvement evaluation activities. List improvements prioritized for implementation. Document potential implementation challenges, improvement impact and effort, and associated priority of individual improvements.
Instructions:
Once the self-assessment activities are complete, evaluate and prioritize the selected improvements for implementation.
Organize and hold an improvement evaluation meeting.
- Prior to a meeting:
- Coordinate schedules and send meeting invites.
- Highlight improvement evaluation concepts and needs
- Request individual review of the Chapter 4 materials.
- Export and share all identified improvements with participants
- Request individual review of improvements, identifying no more than five implementation priorities.
- Evaluate individual responses to identify the group’s priorities and prepare to facilitate a focused improvement evaluation discussion.
- During the meeting:
- Use the TAM Data Assistant.
- Navigate improvements, facilitating prioritization, impact vs. effort, and challenge identification and discussion.
- Capture group consensus and evaluation notes or context.
- After the meeting:
- Summarize and share meeting outcomes.
- Identify action items resulting from meeting activities.
After candidate improvements are identified, the next step is to evaluate them, understand effort versus likely payoff, and anticipate implementation challenges. This evaluation step is important for setting priorities and developing a comprehensive improvement strategy.
The TAM Data Assistant allows you to sort, filter, and review a list of improvement identified during the assessment process. Through this interface, you can track evaluation results based on the criteria described below.
Each improvement should be evaluated in the context of other selected improvements, allowing the relative impact, effort, and priority of each improvement to be established (as High, Medium, or Low) with respect to the other identified options. Improvement specific challenges can also be identified for consideration during strategy development.
Impact is characterized by the extent to which new or existing practices will transform TAM related business practices.
Effort is characterized by the level of resources and staff time required and the extent to which those can be incorporated into the responsibilities and budgets of existing business units.
Priority is established on the basis of when that improvement would be targeted for implementation, ranging from immediate action to being recognized for future, unplanned action.
Challenges can be categorized as into distinct categories of Time, Resource, Expertise, Coordination, Change, or Other.
Conceptual examples illustrating the application of these evaluation factors are provided on the following pages.
TAM Data Assistant Uses
The TAM Data Assistant provides functionality for recording ratings of impact, effort, priority and challenges for each selected candidate improvement.
Additional Recommendations
An iterative approach to improvement evaluation is recommended. To the extent practical, this process should also involve external stakeholders and external planning processes. For example, the goals and objectives stated in agency strategic plans should be incorporated into prioritization of improvement action.
The availability, workload, and resources of impacted business units should also be considered, as well as the engagement and enthusiasm for change found in potential project sponsors and business leads. Without stakeholder engagement, it is unlikely that a data or information system improvement will be successfully and sustainably implemented within routine business.
Improvement Evaluation Tools
The figure below demonstrates the TAM Data Assistant functionality supporting improvement evaluation. Highlighted are five key aspects of this interface:
- Sort and Display Functionality – organize improvements identified during the self-assessment process
- Filter Functionality – apply criteria to filter the improvements based on Area, Challenge, Priority, Effort, Impact, as well as other factors.
- Individual Improvement Details – see details for each selected improvement
- Evaluation Criteria – establish the improvement’s impact vs. effort, priority, and associated challenges.
- Assessment Information – review the current and desired state of the associated element, as well as provide a link to quickly return to, and adjust, the associated assessment information.
- Priority: The Low, Medium, High priority value assigned to the improvement.
- Impact: The Low, Medium, High impact value assigned to the improvement.
- Effort: The Low, Medium, High impact value assigned to the improvement.
- Time Challenge: An indicator of whether a time challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified).
- Resource Challenge: An indicator of whether a resource challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified)
- Expertise Challenge: An indicator of whether an expertise challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified)
- Coordination Challenge: An indicator of whether a coordination challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified)
- Change Challenge: An indicator of whether a change challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified)
- Other Challenge: An indicator of whether another type of challenge was identified for the improvement (0 = no challenge was identified, 1 = a challenge was identified)
- Status: An indicator of whether the improvement was or was not selected for improvement.
- Evaluation Notes: Improvement notes captured during the self-assessment activity.
The export file can be used to readily list, filter, sort, and apply calculations which may be helpful in communicating the priorities for improvement. By joining these results with the detailed assessment information, the user can further refine the priorities for improvement.
TAM Data Assistant Quick Reference Guide
For more detailed information on the TAM Data Assistant’s functions and use, see the TAM Data Assistant Quick Reference Guide.
Transforms current business in a way that addresses major process pain points, is likely to extend to multiple business units, and adds value to multiple business processes.
Medium ImpactMakes existing business processes significantly more efficient and effective, however may be within a limited area of business (e.g. a specific business function or process area).
Low ImpactContributes a minor adjustment to an existing business process, but will not significantly change the business. In general, these improvement may already be informally in place, but are simply being formalized or being made clearer in the context of the program at large.
Requires a major commitment of resources and staff time, typically across multiple business units. Examples would include a major IT application, a statewide technology deployment, etc.
Medium EffortMay be incorporated within typical budgets and resources but would require planning and coordination, typically limited to a specific business function or process area.
Low EffortCan be included within the routine responsibilities of a business unit or working group and typically able to be completed within a short timeframe.
Targeted for immediate action.
Medium PriorityDesired to begin within the next several investment or planning cycles (e.g. 1-2 years).
Low PriorityRecognized, but not anticipated for action within the near future and unlikely to be incorporated into near term planning activities.
Recommended when the time available is limited for the extent of the effort.
ResourcesRecommended when level of resources or staff time would require executive approval.
ExpertiseRecommended when the expertise required is not available to the DOT without specialized support.
CoordinationRecommended when engagement and agreement is required across many different areas of business within the DOT, particularly when many of the impacted business units do not typically work together as part of the routine business of the agency.
ChangeRecommended when the improvement will significantly transform current business across multiple business units and processes, requiring extensive process reengineering and/or training to those impacted.
Filter for High Impact, Low Effort improvements. Consider improvement opportunities which deliver significant value without substantial effort. Where practical for immediate investment, communicate these “low hanging fruit” to decision-makers as easy wins.
Combine Assessment and Improvement InformationCombine assessment and improvement information using the Element ID field. Use this to improve communication of improvement priorities by also relating current or desired performance.
Improvement of Low Performing ElementsUse the combined assessment and improvement information to sort the improvement list for low performing elements (“increasing” by assessment current level). Identify improvements to lowest performing elements.
A low performing element may not always stand on its own as a priority of the organization, but also consider the interrelated nature of performance within the framework. Lagging performance in one aspect of performance can impact ability to be successful in other areas.
Improvement of Elements with Large Performance GapsUse the combined assessment and improvement information to sort the improvement list based on performance gaps (“decreasing” by the different between desired and current level). Identify improvements in the high performance gap elements.
Consider whether initial improvements in these areas should be prioritized, given that multiple improvements over an extended period of time will likely need to be implemented.