Undergraduate Program Continuous Improvement Policy

The Undergraduate Program Continuous Improvement Policy specifies the procedures that the Department follows to ensure continuous improvement of its undergraduate degree programs. The Department recognizes that in order to provide a high quality program that meets the needs of its undergraduates it must make use of multiple forms of feedback and assessment to collect the data it needs to evaluate the current status of the program and to make sound recommendations for improvement.

To guide this process, the Department has established a set of program outcomes and program objectives for the BS degree program. Outcomes are statements made about the skills and capabilities of senior Computer Science majors at the time they graduate. Objectives are statements made about the skills and capabilities of Computer Science alumni approximately three to five years after they graduate. The truth of these assertions is established via the assessment and evaluation mechanisms described by this policy.

The process used by the Department includes the following:

• tracking feedback for and changes made to individual courses

• tracking ratings of course instructors

• tracking survey feedback provided by students, alumni and employers

• collecting and reviewing examples of student coursework

This information is then used by the Undergraduate Committee to make changes in the program to address problems and improve quality.

• Course Assessment Policy
  • Course Coordinator
  • Course Profile
  • Faculty Course Questionnaires
  • Course Assessment Document
  • Course Dossier
• Surveys
• Independent Review Board

Course Assessment Policy

All undergraduate Computer Science courses (with the exception of special topics courses, independent study courses, and courses for non-majors) are required to implement the following policies related to course assessment:

• Course Coordinator
• Course Profile
• Faculty Course Questionnaires
• Course Assessment Document
• Course Dossier

Course Coordinator

Each undergraduate course will be assigned a course coordinator tasked with the goal of managing the course, setting course-related policies, establishing course objectives and learning goals, and ensuring consistency across different instances of the course. In addition, the course coordinator is responsible for following the policies stated here to ensure that changes to the course are applied in response to continuous improvement data. The specific responsibilities of a course coordinator are to

1. maintain the course profile
2. review course FCQ data
3. maintain the course assessment document
4. manage collection of course material used for program assessment

The Undergraduate Committee is in charge of maintaining this list of responsibilities and may make changes to it on a semester-by-semester basis.

Course Profile

The course profile includes the following information:

• course metadata (title, number, credit hours, coordinator, URI)
• catalog description
• textbook
• references
• list of instructors for the last 3 years
• meeting times
• course outcomes
• relationship of course outcomes to program outcomes
• prerequisites by topic
• major topics covered
• assessment plan for the course
• relationship to continuous improvement process
• curriculum category content by credit hour

The course profile is used to track important information about a course and represents a contract as to what is taught in the course and what the Department can expect from students that successfully complete the course. The profile also provides information about a course that is needed during reviews of the Department by ABET (Accreditation Board for Engineering and Technology).

The Undergraduate Committee must review any changes to a course profile before these changes become official. This review ensures that consistency across courses is maintained and that any changes related to course outcomes do not adversely affect courses dependent on those outcomes.

Faculty Course Questionnaires

Faculty Course Questionnaires (FCQs) are administered by the University every term for each instructor of each class taught. FCQs allow students to rate their instructors and courses and provide information to instructors for teaching improvement, to students for course selection, and to deans and chairs for management. Detailed FCQ results are available on the CU Faculty Course Questionnaire website, while summaries of FCQ results for Computer Science courses are available at Computer Science Faculty Course Questionnaires.

The course coordinator must review the FCQ results for the course as soon as the results are available each time the course is taught. FCQ results are usually available near the beginning of the following semester. The review must include

• reading comments submitted by students of the course

• writing a high-level summary of student comments

• highlighting relevant criticism of the course

• discussing what changes (if any) will be made to the course in response

The results of this review will be documented in the Course Assessment Document.

Course Assessment Document

The course coordinator must update the Course Assessment Document of a course during the semester after an instance of the course is taught. This update should occur after the FCQ review is created as that review must be stored in the Course Assessment Document. In addition to the FCQ review, the course coordinator must discuss the performance of the class for each of the course objectives and provide evidence for this assessment.

As an example, if a course outcome is that

"students will know how to create UML class diagrams to document the design of a software system"

then the instructor must state what percentage of the students mastered this outcome and tie this assessment to evidence that backs the statement. For instance, for the previous outcome, an instructor might say

"95% of the students in my course mastered this outcome as evidenced by their performance on question 5 of the final. That question is reproduced below and 95% of the class received full credit for this question."

After discussing each course outcome, the coordinator must discuss the implications of these individual assessments on the course overall. For instance, if students are struggling to achieve certain outcomes, what changes will the instructor make to improve student performance the next time the course is taught?

The Course Assessment Document will be stored online in a manner that allows faculty to review previous versions of the document and to edit and/or upload new versions of the document each time the course is taught. The Undergraduate Committee is responsible for reminding faculty to update these documents in a timely fashion and to track the response rate until all updates for the previous semester are complete.

Course Dossier

To assist in the Department's goal of improving the quality of its undergraduate program, the Undergraduate Committee is adopting a uniform policy with respect to the collection of course materials from undergraduate courses into course dossiers. These dossiers will be used in conjunction with the Course Assessment Document to provide tangible, external evidence that can be evaluated to assess the quality of each undergraduate course. This information is also useful for maintaining high-quality course dossiers that must be presented to ABET reviewers during an ABET site visit.

Course dossiers consist of the following items:

• course profile

• course assessment document

• instructions for labs/demos conducted in class

• examples of graded assignments/lab assignments (with examples of student work from low, medium and high performing students and student identities concealed)

• examples of graded quizzes and exams, including mid-terms and finals (with examples of student work from low, medium and high performing students and student identities concealed)

• copies of course materials used in instruction (e.g., workbooks, textbooks, handouts, etc.)

• task force reports resulting from course reviews (if applicable)

The course coordinator is tasked with acquiring representative examples of the above material and providing them to the Main Office Administrator in ECOT 717 for online storage each time the course is taught:

• An instructor should bring hard copy materials related to a course dossier to the Main Office Administrator for scanning and storage online.

• Textbooks will not be scanned but should be retained by the instructor so the appropriate text can be provided to ABET reviewers during future site visits.

• Graded material should be scanned by the Main Office Administrator before the material is returned to the students.

• Materials that are in electronic form should be organized into an appropriate file hierarchy, archived in a compressed zip or tar file, and delivered to the Main Office Administrator to be stored online.

This should be done anytime course material changes significantly. Significant changes, of course, should be documented in the Course Assessment Document either in the semester before the change is made or, at the latest, in the semester after the change occurred.

Senior Thesis and project team-based courses have some additional guidelines for creating the course dossier:

Senior Thesis

The final thesis document of an undergraduate senior thesis must be submitted to the Main Office Administrator in electronic form to be stored online.

Project Team-Based Courses

In project team-based courses, artifacts such as documents and source code often undergo multiple changes throughout the semester. Rather than collecting each version of each artifact, the course coordinator should collect only the completed project for each team for submission to the Main Office Administrator. Projects completed by three project teams -- representing low, medium, and high performing teams -- should be submitted. The course coordinator should make a best effort to conceal the identity of the team members with the understanding that this can sometimes be impractical given the size and nature of the material.

The Undergraduate Committee is responsible for coordinating with the Main Office Administrator to ensure that materials are collected for each instance of an undergraduate course and to assist in the process of collecting material from each course that was taught during the previous semester. In particular, a reminder asking for course materials will be sent at the end of the last week of classes. A second reminder will be sent at the start of the first week of classes of the next semester. Finally, a third reminder will be sent starting on the third week of the semester after a course has been taught, with weekly reminders sent until the required materials have been acquired.

Surveys

Each year, the College of Engineering and Applied Science administers a number of surveys to collect information from various College stakeholders -- students, alumni and employers -- which can be used to assess the quality of the College's departments and programs. These surveys include the

• Freshman Survey -- administered to students at the end of their first year in the College

• Senior Survey -- administered to students in the month before they graduate

• Post-Graduate Survey -- administered to students roughly six months after they graduate

• Alumni Survey -- administered to alumni three to five years after they graduate

• Internship Survey -- administered to students who take internship positions with companies during the Summer

• Employer Survey -- administered to companies that hire our graduates

The Undergraduate Committee will make use of the data from these surveys as one point of independent feedback concerning the quality of the BS degree program and whether or not the goals of the program as stated by the program outcomes and program objectives are being met.

In particular:

• The Undergraduate Committee will review the results of the Freshman, Senior, Post-Graduate, and Alumni surveys on an annual basis and transform the knowledge gained into recommendations on how the BS degree program can be improved.

• This information will be captured in the Department's annual assessment report and will also be submitted to the Chair of the Department and the Executive Committee for review and guidance on how to prioritize the various recommendations made.

• The Undergraduate Committee will also review the questions asked on these surveys and recommend updates to the College in time for the updated questions to appear on the next instance of each individual survey.

Independent Review Board

In lieu of an exit exam, the Department will have aspects of its undergraduate program reviewed annually by an independent review board, typically comprised of members of the Department's Advisory Board and other qualified external reviewers. The board's job will be to provide external evaluation and feedback on the Department's goal of continuously improving its undergraduate program.

• The independent review board will be asked to review examples of student work from a selection of undergraduate classes drawn from foundation, track foundation and core, and capstone classes. The board will be asked to evaluate the work and to indicate evidence of our students demonstrating skills associated with the BS degree program outcomes.

• The Chair of the Undergraduate Committee will be responsible for working with the Chair of the Department to schedule time during the annual Advisory Board meeting for the independent review board to meet and review the student work for that year's selection of courses. The Chair of the Undergraduate Committee is also responsible for selecting the work the review board will see, presenting the work to the board, and working with the board to capture their comments and feedback in the Department's annual assessment report.

• The Chair of the Undergraduate Committee will ensure that the review board has reviewed a significant percentage of the undergraduate program by the time of ABET review of the department, which typically occurs every six years after accreditation is received. The student work shown to the independent review board will be drawn from the work gathered as part of the collection of course materials as outlined by the course assessment policy noted above.

• Finally, the Chair of the Undergraduate Committee will be responsible for presenting the "state of the undergraduate program" to the Department's Advisory Board and soliciting their feedback and evaluation of the current program and recommendations for improving the program. These recommendations will also be captured in the Department's annual assessment report. It will be the responsibility of the Undergraduate Committee to review the board's feedback and enact changes in response.