Doing the Program Review
Prepare for the Process
Departments or specific programs for review are identified on the schedule (see Appendix A). The Vice President, Dean, or immediate supervisor of the related area will meet with each department chairperson/director to review the procedure. As part of preparation, each department should establish a Study Team, identify a team leader, and identify a writer. The team leader will usually be the chairperson or director of the department. The Study Team will be responsible for the Program Review process, synthesizing assessment, and preparing the Program Review report. The team leader will interact with the Vice President, Dean, or supervisor on behalf of the department.
State Who You Are and Where You Want to Be
The review process is introspective. During this phase, the department should look back to the prior Program Review and planning documents. The goals, strengths, and weaknesses that were earlier established are important to the process. In addition, the current program plans should be examined. These goals, strengths, and weaknesses can be used to add to the measurable objectives and outcomes throughout the Program Review process. Team members should re-read University, college, division, and departmental Mission Statements to ensure the alignment of vision, mission, goals, planning, and actions.
Mission and Vision Statements. All improvement efforts begin with vision and mission. A vision is a picture of the future the department seeks to create. A statement of "our vision" shows where we want to go, and what we will be like when we get there. Mission is equally important. Whether called a mission or purpose, it represents the fundamental reason for the organization's existence. What are we here to do together (Senge et al, 1994)? In an area's study of its roles and performance, it is useful to review the University's mission statement. Colleges and administrative divisions have developed mission statements as well. Beyond this, it is imperative that each area develops its own mission and vision statements. Copies should be included in the Program Review report. A strong review will show the linkages among the department, college or division, and university mission and vision statements.
Goals that flow from the Mission and Vision. Goals are brief and concise statements of expectations and commitments. They are the purposes to which university, department, and individual efforts are directed. The 'measurability' of the goals should be considered. Goals for which evidence cannot be provided should be restated. Within the goal statements, it is important to consider the stakeholders or customers of the department. Connections should be apparent among university, college, division, and department goals.
Designing measurable outcomes and objectives. Outcomes are statements that describe the department's intent regarding its outputs (be they cognitive, attitudinal, behavioral, products, or services). Outcomes are results oriented. While there is no correct or accepted way of stating outcomes, it is important that the statements are clear and understood by all members of the department (to include the customers). Most importantly, outcomes must be directly measurable.
Plan for Assessment
How many outcomes should be written and assessed? There are likely hundreds of possible outcomes for each department. While the department may feel that it is important to have all of them stated and tied to the processes that are designed to produce the outcomes, it is not expected that each outcome can be assessed each year. Outcomes of the highest priority should be assessed first and others may follow. Over time, all outcomes should be addressed. Nichols (1995) recommends that departments focus on between three to five outcomes each year.
What do some outcome statements look like? Writing outcome statements is not easy but a well-written outcome is fundamental to successful assessment. Given on the next page are good and bad versions of statements. Although they appear very similar, note the differences that add to the specificity and measurability of the outcome.
What are the Program Review Guidelines? Sets of general criteria are provided to guide the departments' review (see the section on HOW THE REVIEWS DIFFER). For academic areas, four criteria have been established that focus on curriculum, resources, faculty, and students. Similarly, several criteria exist for administrative areas that include leadership, information and analysis, planning, human resources, process management, performance results, and student/stakeholder focus and satisfaction. These guidelines should provide a structure to the review and should be addressed. Evidence of quality in each of the guideline areas should be shown. Shortcomings should be described with plans to overcome problems in any of the guideline areas.
Well Written |
Poorly Written |
|---|---|
| Of those graduates seeking employment, at least half will have been offered a job prior to leaving the University. | Graduates will find jobs. |
| Clients will show high ratings of satisfaction with the services received from the office. | All of our clients will be pleased all of the time. |
| Students completing the baccalaureate program will be capable of writing an acceptable journal article and having it published. | Students will be able to write well. |
| Ninety-five percent of all forms will be processed and distributed within 48 hours of their receipt. | A backlog of forms should be avoided. |
| Graduates will demonstrate proficient use of current software packages that relate to their field. | Students will have computer skills. |
| Job backlog will not exceed 4 days without feedback and explanation to the client regarding the status of the job. | We will respond to client requests. |
| Retention rates for the program will exceed the university, SSHE, and National averages. | Our students will show high retention rates. |
| Service to non-traditional students will be rated as high as service to traditional students. | Non-traditional students will not be forgotten. |
| Students' average rating of faculty academic advising will exceed national averages. | Student advising is important. |
Using external consultants. State System of Higher Education of Pennsylvania Board of Governors' policy states that, At least once every 10 years, each academic program shall be subject to review by an appropriate external program consultant either in the course of the university review process or in the context of review by a specialized accrediting agency recognized by the Council on Postsecondary Accreditation (COPA).
While this policy specifically addresses academic programs, the same should hold true for all programs at Shippensburg University. External consultants may provide a number of services to the department under review:
- They could facilitate the Program Review process by assisting in the development or refinement of goals and objectives.
- They could visit the department, evaluate processes and procedures, and submit an external review report that can be incorporated into the final Program Review report.
- They could review the assessment procedures and recommend improvements or alternative assessment.
- They could help to establish benchmarks appropriate for the department or program.
- Write the entire Program Review report.
- Provide a 'glowing' evaluation. Weaknesses are as important as strengths.
- Promote the department's or an individual's agenda.
Use the Data
Where does the data come from? The primary data sources for the department will be from their own assessment and data collection efforts. Additional data should be gathered from university-wide sources provided through the Office of Institutional Research and Planning. It is best if the assessment is on-going and, for the most part, completed well before the time of review. During this phase, data from assessment efforts should be integrated into a reportable format. Data should be collected and organized to respond to each of the measurable objectives and outcomes and be compared to previous and/or external benchmarks. The following question should be answered: "What evidence exists to respond to each outcome and, in turn, each goal, strength, or weakness?"
Matching assessment and data to the outcomes. For each outcome statement, a specific strategy for assessment should exist. The data that are collected may then be reported in response to the outcome. Outcomes for which data exceed the expectations show strengths. Outcomes for which the results do not meet expectations require action. Outcomes for which data do not exist require restating or new assessment planning. Many departments are new to the Program Review process. If assessment has not occurred, an assessment and/or data collection plan should be formatted and reported.
Data as information, knowledge, and evidence. Data alone do not sufficiently speak to quality. It is the analysis, synthesis, and interpretation of the data into information and knowledge that make assessment powerful. A strong Program Review employs information to answer questions about the achievement of goals and outcomes. The Program Review should ultimately address the degree to which the department is meeting its and the university's missions.
Write the Report
Addressing the guidelines. A formal report will be prepared and submitted to the appropriate Vice President, Dean, or supervisor. The report may be organized with corresponding sections to the Program Review Guidelines (detailed later in this report). The following questions may be helpful:
"What are the results of the assessment?"
"How do we know that the objectives and outcomes are or are not being met?"
"Are the programs and/or services adequately delivered?"
"Are students and other stakeholders satisfied?"
Establishing evidence for the goals. The primary role of assessment is to provide data as evidence for outcomes and objectives. These data, either quantitative (numerical) or qualitative (i.e. focus groups, customer discussions, critical incidents), will show if the outcomes have been achieved. Strengths of the area are found when objectives are met. Weaknesses and opportunities for improvement exist when the outcomes and objectives have not been met. Data should be presented as evidence in relation to outcomes, objectives, goals, and mission. In addition, the area's plan for assessing outcomes and objectives should be included in the report. The checklist in Appendix D should be helpful in organizing the report.
Change and improvement. Recommendations for change and improvement should be stated and based on the extent to which the goals and mission of the area have been met. Recommendations for change or increased resource support without evidence tied to objectives present a weak argument. The university, and each department, is showing successful use of the planning process when change is tied to feedback from information and knowledge. Continuous improvement is manifested when the time period between feedback on a process and change of the process becomes shorter.
Follow-up on the Report
What is the Program Review Committee? The evaluation of the review is conducted in an open consultative style by the Review Committee and the department. The Review Committee consists of the Vice President of the area under review (or his/her designee) as chair and at least three other individuals selected by the chair. These individuals should not be part of the department under review. The Director of Institutional Research and Planning serves as ex-officio member of the Committee and may be invited by either the Review Committee or the Study Team to facilitate the process at any point. The Review Committee will function to monitor the outcomes of the process so that the design of the review will allow for individual differences between departments as well as changes within the institution and State System of Higher Education.
How the department is evaluated. The report should highlight each of the goals and objectives and show either congruence or non-congruence. In appropriate cases, the supervisor (Dean, Asst. Vice President, or other manager) should forward a critique of the department and the report to the Review Committee that will critically review the report. Following a meeting of the Review Committee, the Vice President or chair of the Review Committee will prepare a draft of Committee recommendations and share them with the department through the team leader and supervisor (when applicable). Either party may suggest further interaction between the Committee and the department. A Program Review Summary will be prepared by the Vice President and submitted to the President and appropriate State System of Higher Education official. The office of Institutional Research and Planning should receive a copy of the report and summary for all program reviews.
Linkage to University Program Plans. A connection should exist among the goals used for program review and the three-year program planning goals. While the statement of the goals may need to be somewhat different, they should be generally similar. Assessment evidence can therefore be used to support program planning. The need for significant new goals should be well documented and supported by multiple sources of evidence from assessment.
Linkage to Resources. In a time of tighter resources, the use of information to support decision-making is critical. The results from program reviews will help decision-makers allocate resources to maximize institutional effectiveness. In addition, the university will be able to show accountability to stakeholders such as students, their parents, legislature, and other funding organizations.
What goes off campus? A short version of the Program Review Summary will be submitted to the Chancellor's Office in a specified format. A summary of university program review will be included in a system report to the Board of Governors.
What's left to do after the Program Review? Completing the document is not an end to the process. Members of the department should have the opportunity to dialog regarding the findings. Each individual should understand the goals and objectives of the area and incorporate the ideas into their own plans and goals. Building a sense of shared vision within the department will benefit efforts at effectiveness, accountability, and improvement.