Guidelines for Self-Assessment of Graduate Programs in the Biomedical Sciences

Guidelines for Self-Assessment of Graduate Programs in the Biomedical Sciences

A Report of the Task Force on Benchmarks of Success in Graduate Programs AAMC GREAT Group. September 2000

Survey Instruments
Report of the Task Force on Benchmarks of Success in Graduate Programs
AAMC GREAT Group
September 2000

 

ii Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000
Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 Association of American Medical Colleges
To order additional copies of this publication, please contact:
Association of American Medical Colleges
Section for Publication Orders
2450 N Street, NW
Washington, DC 20037-1134
Phone: 202-828-0416
Fax: 202-828-1123
Price: $35.00, plus shipping
(Item Code: GBSN)
AAMC Members: $25.00, plus shipping
(Item Code: GBSM) Association of American Medical Colleges
Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 iii

iii

GREAT Group Task Force on Benchmarks of Success in Graduate Programs
Thomas Fox, Ph.D., Chair,
Associate Dean for Graduate Education,
Harvard Medical School
Roger Chalkley, D.Phil.,
Director, Interdisciplinary Graduate Programs
Vanderbilt University
Brenda Russell, Ph.D.,
Executive Associate Vice Chancellor for Research, University of Illinois-Chicago, College of Medicine
Chair, Associate Dean for Graduate Education,

Harvard Medical School
Roger Chalkley, D.Phil., Director, Interdisciplinary Graduate Programs,
Vanderbilt University
Brenda Russell, Ph.D., Executive Associate Vice Chancellor for Research, University of Illinois-Chicago, College of Medicine

AAMC Staff

 

Allan C. Shipp, Assistant Vice President for Biomedical and Health
Sciences Research
Stephen J. Heinig, Senior Staff Associate, Division of Biomedical and
Health Sciences Research This document was posted to the GREAT Group Web Site <www.aamc.org/members/great/103226/great_selfassessmentreport.htmlgre> in June 1999 as a means to solicit comments as well as to provide electronic access to the survey instruments. For the valuable input received, the Task Force would like to thank the GREAT Group membership and leadership. iv Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Association of American Medical Colleges Table of Contents I. Preamble ..............................................................................................................1

II. The History and Mission of Graduate Education in Medical Schools ................ 2

III. The Goals and Objectives of Successful Graduate Programs................................4

IV. How Programs Can Work Toward Achieving Success..........................................6

Step 1: Defining Program Goals and Objectives ..................................................6

Step 2: Pursuing Activities That Promote Attainment of Goals and Objectives....6

Step 3: Identifying Indicators of Success ..............................................................7

Step 4: Data Gathering ........................................................................................8

Step 5: Implementing Change ..............................................................................9

V. Self-Assessment Surveys......................................................................................10

Surveys:

 

Qualitative Questionnaire for Self-Evaluation of Graduate Programs ................11 Assessment of Faculty and Ph.D. Program Characteristics ................................14 Assessment of Admissions and Graduate Student Populations by Program........18 Survey of Student Career Goals and Satisfaction................................................21 Aggregated Data on Student Career Goals and Satisfaction ..............................23 Faculty Assessment of Recent Graduates............................................................25

Tables:

 

Existing Sources of Data for Graduate Program Self-Assessment Table 1. Assessment of Faculty and Program Characteristics ......................26 Table 2. Assessment of Graduate Student Populations ................................27 Table 3. Assessment of Recent Graduates ....................................................28 Other Resources ................................................................................................29

Association of American Medical Colleges

 

Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 1

1

Self-Assessment of Graduate Programs in the Biomedical Sciences NARRATIVE GUIDE AND COMPANION SURVEY INSTRUMENTS

I. Preamble

 

The Association of American Medical Colleges (AAMC) Group on Graduate Research, Education, and Training (GREAT Group) is a professional body of approximately 300 individuals with a responsibility for, or interest in, Ph.D. training that occurs in the medical school setting. At its 1996 and 1997 conferences, the membership of this group recognized the value of having graduate programs periodically conduct a formal process of self-assessment to discern the extent to which they are meeting their goals. Participants at these meetings also expressed the desire for resources of which program directors could avail themselves when taking on the task of defining goals and identifying the indicators that allow assessment of whether their objectives are being achieved. In light of this, the GREAT Group membership called on its Steering Committee to appoint a task force to develop just such a resource. The Task Force on Benchmarks of Success in Graduate Programs was thus assembled and worked over the following year on this narrative guide. The purpose of the guide is to outline for graduate programs a process by which they can develop and measure their own benchmarks of “success,” however they choose to define it. This guide is not prescriptive and thus does not impose on programs any particular criteria or approach. Rather, it is suggestive, pointing to a wide array of possibilities and outlining a model process that users of this guide should adapt to their own particular circumstances and priorities. A companion to this narrative guide is a set of survey instruments that are attached as appendices. One is a suggested tool to allow programs to conduct qualitative assessments of their own programmatic and educational characteristics. Other surveys, also useful as institutional assessment tools, will have the added benefit of allowing individual programs to compare their own measures against other programs that have used the same or comparable instruments. It is important to note that this project is not a prelude to further accreditation of graduate programs, most of which are already accredited by state, regional, or local accrediting bodies. Nonetheless, the AAMC and the Liaison Committee on Medical Education (LCME) do encourage self-evaluation for the purposes of maintaining the quality of Ph.D. education, and thus this document was conceived of and prepared by the GREAT Group as a resource to its own membership. Input on this guide and other aspects of this project are welcomed and may be sent to the GREAT Group Executive Secretary, AAMC, 2450 N Street, N.W., Washington, D.C. 20037.

this project is not a prelude to further accreditation of graduate

programs, most of which are already accredited by state, regional, or local accrediting bodies. Nonetheless, the AAMC and the Liaison Committee on Medical Education (LCME) do encourage self-evaluation for the purposes of maintaining the quality of Ph.D. education, and thus this document was conceived of and prepared by the GREAT Group as a resource to its own membership. Input on this guide and other aspects of this project are welcomed and may be sent to the GREAT Group Executive Secretary, AAMC, 2450 N Street, N.W., Washington, D.C. 20037.

most of which are already accredited by state, regional, or local accrediting bodies.

 

Nonetheless, the AAMC and the Liaison Committee on Medical Education (LCME) do encourage self-evaluation for the purposes of maintaining the quality of Ph.D. education, and thus this document was conceived of and prepared by the GREAT Group as a resource to its own membership. Input on this guide and other aspects of this project are welcomed and may be sent to the GREAT Group Executive Secretary, AAMC, 2450 N Street, N.W., Washington, D.C. 20037. 2 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Association of American Medical Colleges II. The History and Mission of Graduate Education in Medical Schools

 

For almost a century, medical schools have played an important role in graduate education, a role that was significantly expanded after World War II, when the federal government vested in academic institutions the predominant share of responsibility for conducting the national research effort and for training successive generations of researchers. This arrangement, largely the vision of Vannevar Bush, Director of the Office of Scientific Research and Development under the Roosevelt and Truman administrations, was successful because it integrated teaching and research. This allowed the educational process to benefit from the knowledge emerging from cutting edge scientific activity, while research benefitted from the exuberance, creativity, and effort of young scholars eager to learn and become skilled investigators. This paradigm has produced a highly successful system of training and has led to graduate programs as diverse as the institutions that house them. Many programs, including some in biomedical research, are based within schools of arts and sciences at colleges and universities, an arrangement that integrates graduate training within the broad tradition of scholarship at these institutions and allows undergraduates exposure to the research programs and graduate level activities of departments. While learning from their own mentors, graduate students in turn can develop their own didactic skills in serving as teaching assistants and instructors in baccalaureate-level courses. Medical schools, on the other hand, are responsible for most of the graduate training that occurs in the biomedically-related sciences, a system that has grown out of the relationship between basic biomedical research and clinical activity. Research is an iterative process in which the findings of fundamental scientific investigations yield knowledge leading to new, more effective clinical practices, while clinical experience generates more questions that must be answered through basic research. Thus, the presence of a graduate program within the medical school allows doctorate students to appreciate the clinical relevancy of their basic science curriculum. Medical students, in turn, benefit from the presence of an active program of research since the latest research findings can be readily incorporated into their curricula, and they can be exposed to, and in some cases inspired to pursue, research careers. As a consequence of these benefits, medical school-based graduate programs have been highly successful. Ninety-eight percent of U.S. medical schools conduct Ph.D. training. In a 1997 survey of these institutions, 81 schools responded and identified 629 individual departments and programs responsible for graduate education (by extrapolation, the total number may be as high as 800-900). The same survey revealed that a majority—some 60 percent—of Ph.D.s in the biomedically- related sciences, in fact, emerge from medical school-based programs. Graduates of these programs go on to become faculty, industrial researchers, and work in other capacities that serve the biomedical sciences.

biomedically-related sciences, a system that has grown out of the relationship between basic biomedical research and clinical activity. Research is an iterative process in which the findings of fundamental scientific investigations yield knowledge leading to new, more effective clinical practices, while clinical experience generates more questions that must be answered through basic research. Thus, the presence of a graduate program within the medical school allows doctorate students to appreciate the clinical relevancy of their basic science curriculum. Medical students, in turn, benefit from the presence of an active program of research since the latest research findings can be readily incorporated into their curricula, and they can be exposed to, and in some cases inspired to pursue, research careers. As a consequence of these benefits, medical school-based graduate programs have been highly successful. Ninety-eight percent of U.S. medical schools conduct Ph.D. training. In a 1997 survey of these institutions, 81 schools responded and identified 629 individual departments and programs responsible for graduate education (by extrapolation, the total number may be as high as 800-900). The same survey revealed that a majority—some 60 percent—of Ph.D.s in the biomedically- related sciences, in fact, emerge from medical school-based programs. Graduates of these programs go on to become faculty, industrial researchers, and work in other capacities that serve the biomedical sciences. 1 Ammons, S.W. and D.E. Kelly. “Profile of the Graduate Student Population in U.S. Medical Schools.” Academic Medicine, Vol 72, No. 9, September 1997: pp 454-459.

Association of American Medical Colleges

 

Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 3

One way graduate programs ensure their success is to conduct self-assessments. This is a process by which programs regularly revisit, and if necessary redefine, their goals while looking toward indicators that allow them to determine the extent to which those goals are being achieved. The concepts of programmatic and institutional diversity, as alluded to previously, are key. Each program fills a different niche, and programs should accordingly use varying criteria for self-assessment. Some programs may emphasize special features of the didactic experience, whereas others may focus on exposure to specialized research approaches. Some programs excel in producing clinical researchers, others basic researchers. Some are successful at training academics, others industry scientists. Thus, programs must discern their primary goals with regard to these and other objectives, and assess themselves accordingly. This guide is intended to aid programs in that process of self-assessment. The following section identifies goals and objectives commonly valued by graduate programs. Section IV outlines a process by which programs can achieve success that involves goal setting, self-evaluation, and efforts toward improvement. The process of self-assessment is discussed in Section V, and a series of survey instruments and tables designed to facilitate the collection of benchmarking data are provided after the narrative section of this document. The final section points readers to other resources (e.g., articles, books, World Wide Web sites) that can shed further light on ways to conduct self-assessments. 4 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

Association of American Medical Colleges

III. The Goals and Objectives of Successful Graduate Programs

 

As a consequence of the diversity discussed above, graduate programs have different missions, goals, and objectives, and serve various student populations. This diversity lends in part to the success of these programs collectively. Nonetheless, there are some educational goals that virtually all programs aim to achieve. Most program directors agree that they intend to produce graduates who are:

  • Competent, skilled experimentalists,
  • Problem solvers,
  • Critical and independent thinkers, and
  • Expert in their field, with both depth and breadth of knowledge. In addition, programs aim to instill in individuals personal characteristics that are key to professional success in any career, leading to scientists who are also:
  • Leaders,
  • Excellent communicators,
  • Good mentors, nurturers, and teachers,
  • Organized administrators,
  • Exemplars of high ethical standards, and
  • Collaborators and team players.

Finally, there are special characteristics that some programs may wish to see in their graduates, given the program’s special mission or setting. Specific programs may wish to produce graduates who excel in techniques particularly relevant to a given discipline, seek to conduct patient-oriented research, or have other specific qualities, training, or goals. Moreover, various programs will seek to serve students who are primarily from underrepresented minorities, are state or community residents, are internationally competitive, or have other unique characteristics. Definitions of program success should encompass these objectives as well. Graduate deans and faculty judge their success not only by the quality and skills of the graduates they produce, but by the attributes of the program itself. Indeed, some characteristics in particular seem to be common to programs that are successful. In a sense, these are the prerequisites of creating a faculty, a student body, and an environment consistent with excellence in education and the program’s overall objectives. Thus, a successful program is generally one that:

  • Has a faculty who are talented educators and capable researchers — Faculty must be skilled in pedagogic techniques, be excellent communicators, and through research, be current on the latest science in their discipline.
  • Provides adequate resources for teaching and research — Even the most talented and skilled faculty cannot educate effectively without the necessary instructional tools, library facilities, and research infrastructure, to which students also need access as part of the didactic experience.
  • Has a faculty who are talented educators and capable researchers — Faculty must be skilled in pedagogic techniques, be excellent communicators, and through research, be current on the latest science in their discipline.

  • Provides adequate resources for teaching and research — Even the most talented and skilled faculty cannot educate effectively without the necessary instructional tools, library facilities, and research infrastructure, to which students also need access as part of the didactic experience.
  • Provides adequate resources for teaching and research — Even the most talented and skilled faculty cannot educate effectively without the necessary instructional tools, library facilities, and research infrastructure, to which students also need access as part of the didactic experience.
  • Association of American Medical Colleges

    Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 5

    • Conducts quality research programs — Research and education go hand in hand, not only to assure currency and credibility on the part of faculty, but also to expose students to the research process, to conduct their own investigations, and to learn from their mentors.
    • Creates and fosters an intellectual environment and scientific probity — Students develop a sense of scholarship, integrity, and professionalism in large part through the examples set by those around them; thus fostering an environment that strives for the highest standards in all these areas is key.
    • Stimulates and promotes a quest for learning and a drive for creative inquiry — Good scientists are both creative thinkers and life-long learners who thirst for answers to unsolved problems and gaps in their knowledge base.
    • Retains good students and minimizes unnecessary attrition — Another indicator of a program’s success is its ability to retain the best students. Conversely, a large attrition of good students or a high occurrence of academic performance problems is undesirable. These characteristics may be signs that the program is not meeting in some way students’ expectations or not properly identifying applicants with adequate prerequisite training.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Conducts quality research programs — Research and education go hand in hand, not only to assure currency and credibility on the part of faculty, but also to expose students to the research process, to conduct their own investigations, and to learn from their mentors.
    • Creates and fosters an intellectual environment and scientific probity — Students develop a sense of scholarship, integrity, and professionalism in large part through the examples set by those around them; thus fostering an environment that strives for the highest standards in all these areas is key.
    • Stimulates and promotes a quest for learning and a drive for creative inquiry — Good scientists are both creative thinkers and life-long learners who thirst for answers to unsolved problems and gaps in their knowledge base.
    • Retains good students and minimizes unnecessary attrition — Another indicator of a program’s success is its ability to retain the best students. Conversely, a large attrition of good students or a high occurrence of academic performance problems is undesirable. These characteristics may be signs that the program is not meeting in some way students’ expectations or not properly identifying applicants with adequate prerequisite training.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Creates and fosters an intellectual environment and scientific probity — Students develop a sense of scholarship, integrity, and professionalism in large part through the examples set by those around them; thus fostering an environment that strives for the highest standards in all these areas is key.
    • Stimulates and promotes a quest for learning and a drive for creative inquiry — Good scientists are both creative thinkers and life-long learners who thirst for answers to unsolved problems and gaps in their knowledge base.
    • Retains good students and minimizes unnecessary attrition — Another indicator of a program’s success is its ability to retain the best students. Conversely, a large attrition of good students or a high occurrence of academic performance problems is undesirable. These characteristics may be signs that the program is not meeting in some way students’ expectations or not properly identifying applicants with adequate prerequisite training.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Stimulates and promotes a quest for learning and a drive for creative inquiry — Good scientists are both creative thinkers and life-long learners who thirst for answers to unsolved problems and gaps in their knowledge base.
    • Retains good students and minimizes unnecessary attrition — Another indicator of a program’s success is its ability to retain the best students. Conversely, a large attrition of good students or a high occurrence of academic performance problems is undesirable. These characteristics may be signs that the program is not meeting in some way students’ expectations or not properly identifying applicants with adequate prerequisite training.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Retains good students and minimizes unnecessary attrition — Another indicator of a program’s success is its ability to retain the best students. Conversely, a large attrition of good students or a high occurrence of academic performance problems is undesirable. These characteristics may be signs that the program is not meeting in some way students’ expectations or not properly identifying applicants with adequate prerequisite training.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Provides effective and efficient support of students’ needs — Program administration has a responsibility to facilitate graduate education and career development. Services must be offered to students that permit identification of scholarships, access to constructive criticism and feedback, contacts with potential employers and fellowships, career counseling, and the provision of other services that benefit individuals while students or once launched into their careers.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.
    • Is valued by the faculty and central administration — For a program to be successful, it must have a prominent place within the mission of the institution and enjoy political support from the faculty and administration. This characteristic is key to accomplishment and obtaining necessary resources.

    6 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

    Association of American Medical Colleges IV. How Programs Can Work Toward Achieving Success

     

    This document is predicated on the value of self-assessment, and thus review is a key activity in allowing programs to discern whether they are meeting their objectives and maintaining the standards that they have set for themselves or that they must meet to remain accredited. Review is, in effect, a circular process by which goals are defined, activities undertaken to achieve them, outcomes observed, and measures taken to assess progress. Informed by the knowledge that measurements allow, programs then revisit their goals and the process begins anew. Of course, outside entities will also be eyeing the extent to which programs achieve certain goals, including the parent institution, visiting consulting committees, NIH training grant reviewers, accrediting bodies, and the authors of independent reputational surveys published in the popular press to assist prospective students in the selection of programs. The presence of rigorous external review enhances, rather than diminishes, the value of self-assessment. The process of self-assessment will inevitably lead to improvements that will benefit the program when these external evaluations are conducted. What follows is a general outline of the process of self-assessment that individual programs may choose to undertake.

    Step 1: Defining program goals and objectives — Programs must first consider the goals and objectives that they are trying to meet. Some of these may be universal in nature—those common to the educational and research missions that all programs share—and some may be specific to the particular niche that the program fills, as described earlier. The goals and objectives should relate to the type of student that the program hopes to attract, the training environment the program wishes to create, and the product the program intends to generate in terms of researcher competencies and career paths. The desired characteristics of faculty and the training environment should also be articulated.

    Defining program goals and objectives — Programs must first consider the goals and objectives that they are trying to meet. Some of these may be universal in nature—those common to the educational and research missions that all programs share—and some may be specific to the particular niche that the program fills, as described earlier. The goals and objectives should relate to the type of student that the program hopes to attract, the training environment the program wishes to create, and the product the program intends to generate in terms of researcher competencies and career paths. The desired characteristics of faculty and the training environment should also be articulated.

    Step 2: Pursuing activities that promote attainment of goals and objectives — Successful programs— those that achieve their goals and objectives—generally undertake a common set of activities to enable their success. These include:

    • Recruiting students who are a good match with its mission — This is achieved in part by communicating effectively to prospective students about the program’s mission and characteristics through literature and other recruitment tools. Providing data on career outcomes of graduates can also enable prospective students to assess if their own aspirations are in line with the achievements of the program’s graduates.
    • Admitting applicants who will do well within the program — Programs use various criteria to identify students that seem to have the preparation and personal attributes that are required for success in graduate school. These may include GRE scores, GPAs, personal essays, undergraduate faculty recommendations, prior research experience and so forth, and the weight given to each criterion will vary depending on the program’s experience and goals.
    • Pursuing activities that promote attainment of goals and objectives — Successful programs— those that achieve their goals and objectives—generally undertake a common set of activities to enable their success. These include:
    • Recruiting students who are a good match with its mission — This is achieved in part by communicating effectively to prospective students about the program’s mission and characteristics through literature and other recruitment tools. Providing data on career outcomes of graduates can also enable prospective students to assess if their own aspirations are in line with the achievements of the program’s graduates.
    • Admitting applicants who will do well within the program — Programs use various criteria to identify students that seem to have the preparation and personal attributes that are required for success in graduate school. These may include GRE scores, GPAs, personal essays, undergraduate faculty recommendations, prior research experience and so forth, and the weight given to each criterion will vary depending on the program’s experience and goals.
    • Recruiting students who are a good match with its mission — This is achieved in part by communicating effectively to prospective students about the program’s mission and characteristics through literature and other recruitment tools. Providing data on career outcomes of graduates can also enable prospective students to assess if their own aspirations are in line with the achievements of the program’s graduates.
    • Admitting applicants who will do well within the program — Programs use various criteria to identify students that seem to have the preparation and personal attributes that are required for success in graduate school. These may include GRE scores, GPAs, personal essays, undergraduate faculty recommendations, prior research experience and so forth, and the weight given to each criterion will vary depending on the program’s experience and goals.
    • Admitting applicants who will do well within the program — Programs use various criteria to identify students that seem to have the preparation and personal attributes that are required for success in graduate school. These may include GRE scores, GPAs, personal essays, undergraduate faculty recommendations, prior research experience and so forth, and the weight given to each criterion will vary depending on the program’s experience and goals.

    Association of American Medical Colleges

     

    Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 7

    • Providing the appropriate training — To ensure the success of enrolled students, a program must provide a comprehensive set of training experiences that not only includes course work and time at the laboratory bench, but that also permits opportunities to attend seminars, to develop communication skills, to observe the professional conduct of mentors and other professionals they may wish to emulate, and to receive appropriate levels of supervision and oversight.
    • Establishing quality control of student progress and performance — Vigilant monitoring of student performance within the framework of programmatic expectations is necessary to ensure that standards are maintained and to identify when intervention may be necessary to correct deficiencies on the part of students.
    • Providing guidance for careers — Once enrolled, students may change career directions as they are exposed to and engaged by new areas of science and professional activity. Mentors and advisors must be poised to work with students to understand the range of possibilities within the career choices they may make and to provide practical advice on achieving their goals.
    • Recruiting effective faculty — The most effective faculty will naturally be those individuals whose talents, research interests, educational skills, and personal values best accommodate the program’s mission. Recruitment efforts should be structured to enhance the yield of applicants with these characteristics. Prior success as a faculty member is key, indicators of which may include history of research support, quality of publications, citation indices, communication and teaching experience, positions previously held, letters of reference, and other assessments.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.
    • Providing the appropriate training — To ensure the success of enrolled students, a program must provide a comprehensive set of training experiences that not only includes course work and time at the laboratory bench, but that also permits opportunities to attend seminars, to develop communication skills, to observe the professional conduct of mentors and other professionals they may wish to emulate, and to receive appropriate levels of supervision and oversight.
    • Establishing quality control of student progress and performance — Vigilant monitoring of student performance within the framework of programmatic expectations is necessary to ensure that standards are maintained and to identify when intervention may be necessary to correct deficiencies on the part of students.
    • Providing guidance for careers — Once enrolled, students may change career directions as they are exposed to and engaged by new areas of science and professional activity. Mentors and advisors must be poised to work with students to understand the range of possibilities within the career choices they may make and to provide practical advice on achieving their goals.
    • Recruiting effective faculty — The most effective faculty will naturally be those individuals whose talents, research interests, educational skills, and personal values best accommodate the program’s mission. Recruitment efforts should be structured to enhance the yield of applicants with these characteristics. Prior success as a faculty member is key, indicators of which may include history of research support, quality of publications, citation indices, communication and teaching experience, positions previously held, letters of reference, and other assessments.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.
    • Establishing quality control of student progress and performance — Vigilant monitoring of student performance within the framework of programmatic expectations is necessary to ensure that standards are maintained and to identify when intervention may be necessary to correct deficiencies on the part of students.
    • Providing guidance for careers — Once enrolled, students may change career directions as they are exposed to and engaged by new areas of science and professional activity. Mentors and advisors must be poised to work with students to understand the range of possibilities within the career choices they may make and to provide practical advice on achieving their goals.
    • Recruiting effective faculty — The most effective faculty will naturally be those individuals whose talents, research interests, educational skills, and personal values best accommodate the program’s mission. Recruitment efforts should be structured to enhance the yield of applicants with these characteristics. Prior success as a faculty member is key, indicators of which may include history of research support, quality of publications, citation indices, communication and teaching experience, positions previously held, letters of reference, and other assessments.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.
    • Providing guidance for careers — Once enrolled, students may change career directions as they are exposed to and engaged by new areas of science and professional activity. Mentors and advisors must be poised to work with students to understand the range of possibilities within the career choices they may make and to provide practical advice on achieving their goals.
    • Recruiting effective faculty — The most effective faculty will naturally be those individuals whose talents, research interests, educational skills, and personal values best accommodate the program’s mission. Recruitment efforts should be structured to enhance the yield of applicants with these characteristics. Prior success as a faculty member is key, indicators of which may include history of research support, quality of publications, citation indices, communication and teaching experience, positions previously held, letters of reference, and other assessments.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.
    • Recruiting effective faculty — The most effective faculty will naturally be those individuals whose talents, research interests, educational skills, and personal values best accommodate the program’s mission. Recruitment efforts should be structured to enhance the yield of applicants with these characteristics. Prior success as a faculty member is key, indicators of which may include history of research support, quality of publications, citation indices, communication and teaching experience, positions previously held, letters of reference, and other assessments.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.
    • Training to improve the pedagogical process — Once having recruited the right faculty, programs have a responsibility to assure their continued success as teachers. Assessment of faculty pedagogical skills and offering opportunities for faculty to improve their abilities as educators is a fundamental responsibility of any program.

    Step 3: Identifying indicators of success — Keeping in mind what the program is trying to achieve, program leaders should consider what factors may serve as reasonable indicators of program success. Many of them have been identified earlier in this guide and include quantitative measures that reflect both the background and preparation of individuals prior to becoming affiliated with the program (e.g., GRE scores, GPAs, etc.), as well as their achievements during and after. Other indicators may correspond to qualitative considerations, such as student satisfaction, producing scientists with a potential for leadership, creating a nurturing environment, and so forth. The process of identifying both types of indicators may involve faculty and student representatives, and as well as someone with the appropriate expertise to validate the correlation between the factors being measured or studied and the program characteristic that is being assessed.

    Identifying indicators of success — Keeping in mind what the program is trying to achieve, program leaders should consider what factors may serve as reasonable indicators of program success. Many of them have been identified earlier in this guide and include quantitative measures that reflect both the background and preparation of individuals prior to becoming affiliated with the program (e.g., GRE scores, GPAs, etc.), as well as their achievements during and after. Other indicators may correspond to qualitative considerations, such as student satisfaction, producing scientists with a potential for leadership, creating a nurturing environment, and so forth. The process of identifying both types of indicators may involve faculty and student representatives, and as well as someone with the appropriate expertise to validate the correlation between the factors being measured or studied and the program characteristic that is being assessed. 8 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

    Association of American Medical Colleges

    Step 4: Data gathering — Once a program has identified appropriate indicators of success, it will then have to obtain data on and assess the relevant characteristics of students, faculty, and other program elements as appropriate. Some indicators that programs may examine are identified in Tables 1 through 3 at the end of this document, grouped according to the aspect of the program being assessed. These tables focus specifically on assessment of faculty and program characteristics, graduate student populations, and recent graduates and their career outcomes. Shown in these tables are sources of these data within the institution as well as external sources that allow comparison to national statistics, where available. It should be noted that these tables focus largely on programs’ characteristics that are inherently quantifiable. Some characteristics of successful programs are highly qualitative in nature, aspects of which can nonetheless be assessed through quantitative means, such as the use of psychometric tools. Such qualitative characteristics include student satisfaction, environmental characteristics, integrity among student and faculty, and so forth. Some possible institutional mechanisms for assessing these characteristics include:

  • Surveys of students — with regard to the quality of faculty, the quality of their education, career goals, and the program environment. These results are potentially valuable on their own, or can be compared with similar surveys in the popular press or at other institutions.
  • Surveys of faculty — with regard to their teaching objectives, the quality of the student body, and the availability of needed resources for teaching and research.
  • Self-evaluation questionnaires — these are targeted at the program directorship and serve to make leaders conduct critical evaluations of the state of their own programs.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Data gathering — Once a program has identified appropriate indicators of success, it will then have to obtain data on and assess the relevant characteristics of students, faculty, and other program elements as appropriate. Some indicators that programs may examine are identified in Tables 1 through 3 at the end of this document, grouped according to the aspect of the program being assessed. These tables focus specifically on assessment of faculty and program characteristics, graduate student populations, and recent graduates and their career outcomes. Shown in these tables are sources of these data within the institution as well as external sources that allow comparison to national statistics, where available. It should be noted that these tables focus largely on programs’ characteristics that are inherently quantifiable. Some characteristics of successful programs are highly qualitative in nature, aspects of which can nonetheless be assessed through quantitative means, such as the use of psychometric tools. Such qualitative characteristics include student satisfaction, environmental characteristics, integrity among student and faculty, and so forth. Some possible institutional mechanisms for assessing these characteristics include:
  • Surveys of students — with regard to the quality of faculty, the quality of their education, career goals, and the program environment. These results are potentially valuable on their own, or can be compared with similar surveys in the popular press or at other institutions.
  • Surveys of faculty — with regard to their teaching objectives, the quality of the student body, and the availability of needed resources for teaching and research.
  • Self-evaluation questionnaires — these are targeted at the program directorship and serve to make leaders conduct critical evaluations of the state of their own programs.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Surveys of students — with regard to the quality of faculty, the quality of their education, career goals, and the program environment. These results are potentially valuable on their own, or can be compared with similar surveys in the popular press or at other institutions.
  • Surveys of faculty — with regard to their teaching objectives, the quality of the student body, and the availability of needed resources for teaching and research.
  • Self-evaluation questionnaires — these are targeted at the program directorship and serve to make leaders conduct critical evaluations of the state of their own programs.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Surveys of faculty — with regard to their teaching objectives, the quality of the student body, and the availability of needed resources for teaching and research.
  • Self-evaluation questionnaires — these are targeted at the program directorship and serve to make leaders conduct critical evaluations of the state of their own programs.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Self-evaluation questionnaires — these are targeted at the program directorship and serve to make leaders conduct critical evaluations of the state of their own programs.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Focus groups — conducted with either students or faculty to allow for identification and articulation of program strengths, weaknesses, and goals. As noted earlier, programs may wish to compare data on their own characteristics against those of other programs. In many cases, however, comparison data may not be current or readily available, may require significant manipulation to be useable, or may not provide the desired type of information about programs and their components. Therefore, it is hoped that the survey instruments found later in this document will help standardize approaches that programs use in collecting this data, facilitating the efforts of individual programs to compare their own measures against those of other programs that are similar in mission or scope. The survey instruments developed for this purpose are discussed in greater length under Section V of this document. Programs are also encouraged to establish their own benchmarks or thresholds based on their prior experience and current goals.
  • Association of American Medical Colleges

     

    Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 9

    Step 5: Implementing change — One important objective of assessing a program’s success is to identify those areas where success, however defined, is not being achieved and to correct the deficiencies or to overcome the obstacles contributing to this situation. Difficulty in achieving success may stem from any number of causes, including: inadequate financial resources, inadequate physical resources, failure to define realistic goals, poor evaluation of applicants, inability to recruit appropriate faculty, undesirable climate or ethics within the program, weak leadership, or many other factors that could be cited. Thus, for assessment to yield a productive outcome, programs must be poised to implement change to allow program success. Implementation of change is not a simple matter and there are a number of issues that programs will have to face in the process:

    • Administrative and financial resources — Implementation of curricular changes, recruitment of new faculty, development of new pedagogical tools, and other improvements will inevitably take resources which may not be readily at hand. Some resources will be financial and others will involve finding the information, skills, and tools necessary for the task. There may also be limitations to change based on what the administration of the parent institution is willing to accept, as well as restrictions placed on programs by state and regional accrediting boards.
    • Faculty involvement — It is also a human characteristic to resist change, and faculty may resist the implementation of new approaches unless there is a careful process of discourse, education, participation, and acceptance. The faculty must be in a position to recognize needed improvement and to accept the means for change, particularly since these measures may involve significant effort on their part.
    • Cross-departmental communications — Many programs are interdisciplinary, involving several departments or even multiple institutions. Challenges related to resources, faculty involvement, and procedural matters may be complicated by these arrangements and by the additional administrative structures involved.
    • Implementing change — One important objective of assessing a program’s success is to identify those areas where success, however defined, is not being achieved and to correct the deficiencies or to overcome the obstacles contributing to this situation. Difficulty in achieving success may stem from any number of causes, including: inadequate financial resources, inadequate physical resources, failure to define realistic goals, poor evaluation of applicants, inability to recruit appropriate faculty, undesirable climate or ethics within the program, weak leadership, or many other factors that could be cited. Thus, for assessment to yield a productive outcome, programs must be poised to implement change to allow program success. Implementation of change is not a simple matter and there are a number of issues that programs will have to face in the process:
    • Administrative and financial resources — Implementation of curricular changes, recruitment of new faculty, development of new pedagogical tools, and other improvements will inevitably take resources which may not be readily at hand. Some resources will be financial and others will involve finding the information, skills, and tools necessary for the task. There may also be limitations to change based on what the administration of the parent institution is willing to accept, as well as restrictions placed on programs by state and regional accrediting boards.
    • Faculty involvement — It is also a human characteristic to resist change, and faculty may resist the implementation of new approaches unless there is a careful process of discourse, education, participation, and acceptance. The faculty must be in a position to recognize needed improvement and to accept the means for change, particularly since these measures may involve significant effort on their part.
    • Cross-departmental communications — Many programs are interdisciplinary, involving several departments or even multiple institutions. Challenges related to resources, faculty involvement, and procedural matters may be complicated by these arrangements and by the additional administrative structures involved.
    • Administrative and financial resources — Implementation of curricular changes, recruitment of new faculty, development of new pedagogical tools, and other improvements will inevitably take resources which may not be readily at hand. Some resources will be financial and others will involve finding the information, skills, and tools necessary for the task. There may also be limitations to change based on what the administration of the parent institution is willing to accept, as well as restrictions placed on programs by state and regional accrediting boards.
    • Faculty involvement — It is also a human characteristic to resist change, and faculty may resist the implementation of new approaches unless there is a careful process of discourse, education, participation, and acceptance. The faculty must be in a position to recognize needed improvement and to accept the means for change, particularly since these measures may involve significant effort on their part.
    • Cross-departmental communications — Many programs are interdisciplinary, involving several departments or even multiple institutions. Challenges related to resources, faculty involvement, and procedural matters may be complicated by these arrangements and by the additional administrative structures involved.
    • Faculty involvement — It is also a human characteristic to resist change, and faculty may resist the implementation of new approaches unless there is a careful process of discourse, education, participation, and acceptance. The faculty must be in a position to recognize needed improvement and to accept the means for change, particularly since these measures may involve significant effort on their part.
    • Cross-departmental communications — Many programs are interdisciplinary, involving several departments or even multiple institutions. Challenges related to resources, faculty involvement, and procedural matters may be complicated by these arrangements and by the additional administrative structures involved.
    • Cross-departmental communications — Many programs are interdisciplinary, involving several departments or even multiple institutions. Challenges related to resources, faculty involvement, and procedural matters may be complicated by these arrangements and by the additional administrative structures involved.

    10 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

    Association of American Medical Colleges V. Self-Assessment Surveys

     

    As previously noted, a companion to this narrative gu
    ide is a set of surveys to allow individual programs to conduct their own self-evaluations and to see where they stand relative to other programs that may have conducted the same data collection efforts. There are four elements to the survey project. The first two surveys collect data at the institutional and program levels. The second set of surveys collect data on the qualities, objectives and satisfaction of individual students within the program. The results of these surveys can be compiled by individual programs and, once aggregated, may yield generalizable information on student populations. The specific survey topics are:

    • Faculty and Ph.D. program characteristics — The faculty are the core of any program and thus pivotal to its success. Consequently, it is worth looking at the numbers, demographics, credentials, and research experience of faculty, in addition to the resources and environment within which the program takes place.
    • Admissions and graduate student populations — Programs should look toward the qualifications, training, research experience, and demographics of incoming students, in addition to their progress as students, and their retention and attrition rates.
    • Student career goals and satisfaction — The student perspective is equally important, particularly as regards their view of faculty, coursework, research environment, and how the program enabled them to meet their career objectives. This document includes a survey for dissemination to individual students in specific graduate programs, as well as another instrument designed to allow reporting of aggregate data for comparison purposes.
    • Recent graduates — Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in the final survey instrument in this report. If programs using these surveys choose to compare data, the goal should not be to assess relative “quality” based on any single set of data. In fact, there is no reason to believe that, on any single criterion, one value is inherently better than another. Having a high faculty-to-student ratio, for example, is not inherently a “good” characteristic if faculty devote little time to students and teaching. So each variable will have to be assessed in the context of a particular program’s setting and goals. The surveys are found in the following section of this document. Individual programs should modify them in whatever way best meets their internal assessment needs.
    • Faculty and Ph.D. program characteristics — The faculty are the core of any program and thus pivotal to its success. Consequently, it is worth looking at the numbers, demographics, credentials, and research experience of faculty, in addition to the resources and environment within which the program takes place.
    • Admissions and graduate student populations — Programs should look toward the qualifications, training, research experience, and demographics of incoming students, in addition to their progress as students, and their retention and attrition rates.
    • Student career goals and satisfaction — The student perspective is equally important, particularly as regards their view of faculty, coursework, research environment, and how the program enabled them to meet their career objectives. This document includes a survey for dissemination to individual students in specific graduate programs, as well as another instrument designed to allow reporting of aggregate data for comparison purposes.
    • Recent graduates — Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in the final survey instrument in this report. If programs using these surveys choose to compare data, the goal should not be to assess relative “quality” based on any single set of data. In fact, there is no reason to believe that, on any single criterion, one value is inherently better than another. Having a high faculty-to-student ratio, for example, is not inherently a “good” characteristic if faculty devote little time to students and teaching. So each variable will have to be assessed in the context of a particular program’s setting and goals. The surveys are found in the following section of this document. Individual programs should modify them in whatever way best meets their internal assessment needs.
    • Admissions and graduate student populations — Programs should look toward the qualifications, training, research experience, and demographics of incoming students, in addition to their progress as students, and their retention and attrition rates.
    • Student career goals and satisfaction — The student perspective is equally important, particularly as regards their view of faculty, coursework, research environment, and how the program enabled them to meet their career objectives. This document includes a survey for dissemination to individual students in specific graduate programs, as well as another instrument designed to allow reporting of aggregate data for comparison purposes.</li> <li>Recent graduates — Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in the final survey instrument in this report. If programs using these surveys choose to compare data, the goal should not be to assess relative “quality” based on any single set of data. In fact, there is no reason to believe that, on any single criterion, one value is inherently better than another. Having a high faculty-to-student ratio, for example, is not inherently a “good” characteristic if faculty devote little time to students and teaching. So each variable will have to be assessed in the context of a particular program’s setting and goals. The surveys are found in the following section of this document. Individual programs should modify them in whatever way best meets their internal assessment needs.
    • Student career goals and satisfaction — The student perspective is equally important, particularly as regards their view of faculty, coursework, research environment, and how the program enabled them to meet their career objectives. This document includes a survey for dissemination to individual students in specific graduate programs, as well as another instrument designed to allow reporting of aggregate data for comparison purposes.
    • Recent graduates — Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in the final survey instrument in this report. If programs using these surveys choose to compare data, the goal should not be to assess relative “quality” based on any single set of data. In fact, there is no reason to believe that, on any single criterion, one value is inherently better than another. Having a high faculty-to-student ratio, for example, is not inherently a “good” characteristic if faculty devote little time to students and teaching. So each variable will have to be assessed in the context of a particular program’s setting and goals. The surveys are found in the following section of this document. Individual programs should modify them in whatever way best meets their internal assessment needs.
    • Recent graduates — Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in the final survey instrument in this report. If programs using these surveys choose to compare data, the goal should not be to assess relative “quality” based on any single set of data. In fact, there is no reason to believe that, on any single criterion, one value is inherently better than another. Having a high faculty-to-student ratio, for example, is not inherently a “good” characteristic if faculty devote little time to students and teaching. So each variable will have to be assessed in the context of a particular program’s setting and goals. The surveys are found in the following section of this document. Individual programs should modify them in whatever way best meets their internal assessment needs.
  • Association of American Medical Colleges

    Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 11

    11 Survey QUALITATIVE QUESTIONNAIRE FOR SELF-EVALUATION OF GRADUATE PROGRAMS

    Survey Qualitative Questionnaire for Self-Evaluation of Graduate Programs

    External program review should generally be preceded by a process of self-evaluation in which the directorship examines the goals, strengths, weaknesses, and other features of the program. This kind of qualitative examination can be helpful in preparation for externally conducted evaluations as well as in providing points of focus for more detailed, quantitative self-assessments that might follow. Thus, this questionnaire is a tool for program leaders to use for their own internal purposes in reflecting on the mission, organization, administration, and educational processes that are characteristic of the program.

    Mission and Organization

    1. What are your program’s goals and objectives and how were these determined?
    2. In terms of teaching and research, what are the program’s greatest strengths and greatest current needs? Are there important areas of the field not adequately represented in your department?
    3. How do the program’s strengths and weaknesses affect the graduate curriculum and laboratory training?
    4. What is the nature of research funding in your program (e.g., what share is hard money or soft?)?
    5. How does your graduate program substantially differ from other programs in major universities?

    12 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

    Survey Association of American Medical Colleges Educational Process

    1. What is the process by which you revise your curriculum? By whom? Do students have input?
    2. When was the last time your curriculum offering and requirements at the graduate level were reviewed? How were curriculum changes implemented?
    3. Describe your admission process. How do you determine the target number of students to recruit?
    4. What faculty advising and mentoring is provided to graduate students in the first year and dissertation stages?
    5. Is there any mechanism for helping faculty become better mentors?
    6. What are the teaching obligations of faculty?
    7. How do your students acquire skills for professionalism, ethical standards, and career development? Do you have a formal course on these subjects?
    8. If you could change one thing about your graduate program, what would it be if (a) no more resources were available, and (b) if you had new funds to achieve significant change?
    9. What is your student disciplinary mechanism? What are the mechanisms for dealing with violations of academic and research integrity? How is the student represented in the process?

    Association of American Medical Colleges

    13 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 13

    1. Do your students have formal channels of communication with the faculty? Is there a student organization?
    2. What opportunities do students have to teach or to conduct lectures? To what degree are students supported on teaching assistantships?
    3. Do you have a student exit interview? If so, how does it influence your program?
    4. Do you track your graduates in their careers? How do you track them?

    Program administration

    1. What kinds of resources and assistance does the program provide to faculty in carrying out their teaching and research responsibilities?
    2. How does the program, department, or institution reward faculty for effort applied to teaching?
    3. How does the program assist faculty in obtaining external support for their research?
    4. In what other ways does the program administration contribute to the success of its faculty and students?
    5. Survey

      14 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

      Association of American Medical Colleges
      Assessment of Faculty and Ph.D. Programs

      The faculty are the core of any program and thus pivotal to its success. Consequently, it is worth looking at the numbers, demographics, credentials, and research experience of faculty, in addition to the resources and environment within which the program takes place. Graduate deans and others may thus wish to use this questionnaire as a tool to develop data on and assess those characteristics.

      Respondent Information:

      Name of individual providing this data __________________________________________________________________________________
      Title of the person named above __________________________________________________________________________________
      Name of institution to which data pertain __________________________________________________________________________________
      Address __________________________________________________________________________________
      Telephone Fax __________________________________________________________________________________
      E-mail address __________________________________________________________________________________

      Survey

      Association of American Medical Colleges
      Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 15

      Survey ASSESSMENT OF FACULTY AND PH.D. PROGRAMS

      ASSESSMENT OF FACULTY AND PH.D. PROGRAMS

      I. Characteristics of Ph.D.-awarding programs at your medical school

      1. How many programs do you have: _______
      2. How many programs are interdisciplinary: ___________
      3. Do you have an M.D./Ph.D. program? _____Yes _____No

      II. Faculty characteristics

      1. Faculty to student ratios
        1.a. How many graduate students were enrolled in your medical school’s Ph.D. programs as of October of last year:_______
        1.b. How many first-year graduate students participated in your Ph.D. programs as of October of last year:_______
        2.a. How many faculty participated in your graduate programs through teaching, advising, and committees as of October of last year:_______
        2.b. How many basic science faculty participated in your graduate programs through teaching, advising, and committees as of October of last year:_______
        3. Estimate the number of the faculty cited in question II.A.2.a. that were eligible to be thesis advisors to students in your graduate programs: _______
        4. What percentage of the faculty in question II.A.3. have the following number of students in their labs:
        1 _____%
        2 _____%
        3 _____%
        4 _____%
        5 _____%
        >5 _____%
        _______ 100%
      2. Graduate program faculty characteristics
        1. Percent of graduate program faculty who have a primary appointment in a:
        Basic science department _____%
        Clinical science department _____%
        Other _____% ______ 100%
        2. Number and percentage of underrepresented minorities among all graduate program faculty:
        _____ individuals _____%
        3. Number and percentage of women among all graduate program faculty: _____ individuals _____%
        4. Indicators of national and international leadership in the scientific community: Number of graduate program faculty in October of last year who were:
        _____ Officers of scientific societies
        _____ Member of honorary societies (e.g., NAS) and/or recipients of competitive fellowships (e.g., AAAS fellow)
        _____ Scientific review body (e.g., NIH study sections)
        _____ Editorial boards
        _____ International honors
        5. Over the last five years, has the number of your graduate faculty:
        ______ risen
        ______ declined
        ______ stayed the same
      3. Faculty research Number of faculty involved in graduate education (from question II.A.2.a.) who had a grant in the previous 12 months from:
        1. National Institutes of Health ________
        2. National Science Foundation ________
        3. Other federal sources ________
        4. Industrial sources (pharmaceutical, ________ biotechnology, etc.)
        5. HHMI ________
        6. Other private foundations, ________ voluntary health organizations, (e.g., American Heart Association)
        7. Other sources of external ________ research support How many peer-reviewed articles were published in the previous 12 months that were authored by faculty affiliated with your graduate programs? _________

      III. Characteristics of your graduate programs

      1. Training support
        1. National Research Service Award (NRSA)
        a. How many NRSA training grants do you have in all your graduate programs combined? _____
        b. How many NRSA-supported full time training positions (FTTPs) do you have in all graduate programs combined? _____
        c. Do you have an MSTP grant? _____ Yes _____ No
        d. How many FTTPs do your graduate programs support with MSTP funds?_____
        e. How many FTTPs are supported by NRSA fellowships? _____
        2. All sources of support
        a. Pooling all funds for stipend support at your entire medical school (including NRSA), what percent comes from:
        • The NRSA program _____%
        • Research grant assistantships _____%
        • NSF fellowships _____%
        • Other federal sources _____%
        • Teaching assistantships _____%
        • Institutional sources (non-TA) _____%
        • Self (trainee) support _____%
        • Industrial support _____%
        • Foreign support _____%
        • Other non-profit (i.e., foundations _____%
        • and voluntary health organizations such as HHMI, AHA, etc.)
        • Other characteristics
          1. a. Stipends: Do the graduate programs at your medical school offer a minimum stipend for all students? Yes _____ No _____
          b. If yes, what is the amount? ______
          2. Do you offer tuition remission? Yes _____ No _____
          3. Benefits Health Maternity Dental Other a. Do graduate students at your medical school have access to the following benefits (check all that apply): b. For graduate students, does the institution contribute to premiums for the following benefits (check all that apply): c. Do dependents of graduate students at your medical school have access to the following benefits (check all that apply): d. For dependents, does the school contribute to premiums for the following benefits (check all that apply): 4. Are graduate programs at your medical school accredited? Yes_____ No _____
          If so, by what body (bodies) _______________________________________ _____________________________________________________________
        • Assessment of Admissions and Graduate Student Populations by Program

          This questionnaire should be disseminated to each of the graduate doctorate-granting programs within your medical school to assess the qualifications, training, research experience, and demographics of incoming students, in addition to their retention and attrition rates.

          Respondent Information:

          Ph.D. only M.D./Ph.D.

          A. 1. Percent of students who left the program in the past year without completing all degree requirements:3

          3

          ______% ______% 2. Percent of students who left in the past five (5) years without completing all degree requirements:4

          4

          ______% ______%

          __________________________________________________________________________________ Name of department or program to which this survey applies
          __________________________________________________________________________________ Name of person responsible for this survey on behalf of the medical school
          __________________________________________________________________________________ Address
          __________________________________________________________________________________ Address continued
          __________________________________________________________________________________ Telephone Fax
          __________________________________________________________________________________ E-mail address

          ASSESSMENT OF ADMISSIONS AND GRADUATE STUDENT POPULATIONS BY PROGRAM

          Survey I. Incoming Student Populations

          For students who matriculated into the program named above in the fall of the most recent year for which data are available, please provide the following data:

        1. GRE scores (for the fall of the most recent year)
        2. Are GRE scores required as a criterion for determining admissions into this program? ______Yes ______ No
          If yes, please provide the following data:
          Ph.D. only M.D./Ph.D. 1. Verbal: Median1 score for matriculating class: ______ ______
          2. Verbal: Minimum score for matriculating class: ______ ______
          3. Quantitative: Median score for matriculating class: ______ ______
          4. Quantitative: Minimum score for matriculating class: ______ ______
          5. Analytical: Median score for matriculating class: ______ ______
          6. Analytical: Minimum score for matriculating class: ______ ______

          li>
        3. Undergraduate GPAs (for the fall of the most recent year)
        4. 1. Mean GPA in science-related courses for matriculating class: ______ ______
        5. Diversity (as of the fall of the most recent year)
          1. Total number of matriculating students ______ ______
          2. Percentage of women in matriculating class: ______ % ______ %
          3. Percentage of underrepresented minorities: ______ % ______ %
          4. Percentage of students who are US citizens or permanent residents: ______ % ______ %
        6. Acceptance of offers
          1. Number of completed applications to the program: ____________
          2. Approximate percent of applicants who received offers of admission in the fall of the most recent year (not including M.D./Ph.D. students): ______ % ______ %
          3. Approximate percent of those offered admission who actually matriculated in the fall of the most recent year (not including M.D./Ph.D. students): ______ % ______ %
          1 score for matriculating class: ______ ______

          2. Verbal: Minimum score for matriculating class: ______ ______
          3. Quantitative: Median score for matriculating class: ______ ______
          4. Quantitative: Minimum score for matriculating class: ______ ______
          5. Analytical: Median score for matriculating class: ______ ______
          6. Analytical: Minimum score for matriculating class: ______ ______

        7. Undergraduate GPAs (for the fall of the most recent year)
          1. Mean GPA in science-related courses for matriculating class: ______ ______
        8. Diversity (as of the fall of the most recent year)
          1. Total number of matriculating students ______ ______
          2. Percentage of women in matriculating class: ______ % ______ %
          3. Percentage of underrepresented minorities: ______ % ______ %
          4. Percentage of students who are US citizens or permanent residents: ______ % ______ %
        9. Acceptance of offers
          1. Number of completed applications to the program: ____________
          2. Approximate percent of applicants who received offers of admission in the fall of the most recent year (not including M.D./Ph.D. students): ______ % ______ %
          3. Approximate percent of those offered admission who actually matriculated in the fall of the most recent year (not including M.D./Ph.D. students): ______ % ______ %

          II. Development as a Scientist

          For students who graduated from your program in the spring of the most recent year for which data are available, please provide the following data:

          1. Time to degree (defined as years [express with one decimal point] registered full-time in graduate school until completion of thesis, subtracting months on leave or in medical school):
            1. Mean time to degree: ____.__ years
          2. Admission to candidacy2
            2.1. Do all programs at your institution admit to candidacy in the same way? _____ Yes ____ No
            1 Median may best reflect the distribution of scores; other calculations may be used as well, such as mean or ranges. The intent is not to place excessive emphasis on standardized testing scores, which can have limited predictive value.
            2 Admission to candidacy means successful completion of all academic requirements preceding thesis and defense. 20 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000
          Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

          Survey Association of American Medical Colleges

          Association of American Medical Colleges

          2. In your program, is admission to candidacy predicated on (check all that apply): _____ satisfactory performance in required courses? _____ demonstration of cumulative knowledge? _____ preparation of an original research proposal?
          3. a. Does your institution impose time limits on reaching candidacy status: _____ Yes ____ No
          b. If yes, within what number of years must candidates reach this status? _____ Years

        10. Degrees conferred
          1. Number of Ph.D. degrees conferred in the previous academic year (summed over 12 months): ______
          2. Median age of these students (in question C.1) at receipt of Ph.D.: ______

        III. Attrition Ph.D. only M.D./Ph.D.

        1. 1. Percent of students who left the program in the past year without completing all degree requirements:3
        2. Reasons given for leaving the program (per III.A.2.):
          Male Female Male Female
          career changes:5 ____% ____% ____% ____%
          relocation: ____% ____% ____% ____%
          poor performance: ____% ____% ____% ____%
          other: ____% ____% ____% ____%
          _________________________________________________________________
          total 100% 100% 100% 100% 5 ____% ____% ____% ____%

          relocation: ____% ____% ____% ____%
          poor performance: ____% ____% ____% ____%
          other: ____% ____% ____% ____%
          _________________________________________________________________
          total 100% 100% 100% 100%

          IV. Other Aspects of the Graduate Student Experience

          1. Do students receive counseling from an advisor in first year? ____Yes ____No
            B. How often at a minimum must graduate students meet with the thesis committee? _____times per year
            C. What percent of the students take an ethics curriculum: ____%
            D. Do you offer a course on career/survival skills? ____Yes ____No
            3 Calculate as number of students who left in the last 12 months divided by all students matriculated in the program. Also include students on leave for greater than two years as of this date.
            4 Calculate as number of students who left in past five (5) years divided by the number matriculating in the same five-year period.
            5 Including decisions to switch to a different degree program or moving from a dual to a single degree program.
          2. Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 21

            Survey of Student Career Goals and Satisfaction

            This survey should be completed by individual students within each graduate program at the medical school to assess their view of the faculty, coursework, research environment, and how the program enabled them to meet their career objectives.

            Respondent Information:

            Today’s date: ____________________________________
            Name of student completing questionnaire (optional): _____________________________________________________________________________
            Name of program in which you are/were enrolled: ________________________________________________________________________________
            Number of years you have been in the program: ______________
            Years Have you graduated? _______ Yes _______ No
            1. Do you plan on pursuing post-doctoral training: ____Yes ____ No
            2. I ultimately plan on a career in (check all that apply): ___ academia (teaching) ___ academia (research) ___ industry ___ other (please indicate: __________________) ___ don’t know
            3. A. In selecting my next professional position the most important considerations will be (rank order from 1 to 4):
            1. ____ professional development and satisfaction
            2. ____ quality of life and family issues 3
            . ____ geographic location
            4. ____ salary level
            B. With regard to my long-term career goals, the most important considerations are (rank order from 1 to 4):
            1. ____ professional development and satisfaction
            2. ____ quality of life and family issues
            3. ____ geographic location
            4. ____ salary level

            Survey SURVEY OF STUDENT CAREER GOALS AND SATISFACTION

            SURVEY OF STUDENT CAREER GOALS AND SATISFACTION

            22 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Association of American Medical Colleges

            4. How would you rate the following characteristics of your graduate experience (1=excellent, 5=very poor): Scale (circle one) Excellent Very Poor
            a. Level, breadth, and content of courses 1 2 3 4 5
            b. Quality of instruction 1 2 3 4 5
            c. Mentoring/career guidance 1 2 3 4 5
            d. Research environment 1 2 3 4 5
            e. Program activities (e.g., seminars) 1 2 3 4 5
            f. Interactions with fellow students 1 2 3 4 5
            g. Thesis advisory committee 1 2 3 4 5
            h. Program administration 1 2 3 4 5
            I. Mechanism for addressing grievances 1 2 3 4 5
            j. Overall level of satisfaction 1 2 3 4 5 5.
            Comments: ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________

            Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 23

            23

            Aggregated Data on Student Career Goals and Satisfaction

            This instrument can be used to aggregate the data from the Survey of Student Career Goals and Satisfaction, which will allow for analyses of program-wide data and comparisons with other programs that have used this instrument.

            Respondent Information:

            __________________________________________________________________________________ Name of person providing these aggregated, program-level data
            __________________________________________________________________________________ Title of the person named above
            __________________________________________________________________________________ Name of program to which data pertain __________________________________________________________________________________ Address
            __________________________________________________________________________________ Address continued
            __________________________________________________________________________________ Telephone Fax
            __________________________________________________________________________________ E-mail address

            Aggregated Responses to Survey on Student Career Goals and Satisfaction

            1st and 2nd 3rd and 4th Students in the 5th Year Students Year Students Year and Beyond
            1. Percent of students planning on postdoctoral training _____% _____% _____%
            2. Percent of students planning a career in:
            A. academia (teaching) _____% _____% _____%
            B. academia (research) _____% _____% _____%
            C. industry _____% _____% _____% D. other _____% _____% _____% E. don’t know _____% _____% _____%
            3. A. Average ranking (to tenth decimal; e.g., 2.6) of considerations for respondent’s next professional position.
            1. professional development and satisfaction _______ _______ _______
            2. quality of life and family issues _______ _______ _______
            3. geographic location _______ _______ _______
            4. salary level _______ _______ _______
            B. Average ranking (to tenth decimal; e.g., 2.6) of considerations for respondent’s long-term career goals
            1. professional development and satisfaction _______ _______ _______
            2. quality of life and family issues _______ _______ _______
            3. geographic location _______ _______ _______
            4. salary level _______ _______ _______

            Survey

            Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 25

            Survey FACULTY ASSESSMENT OF RECENT GRADUATES

            Survey Faculty Assessment of Recent Graduates

            Faculty assessments of the intellect, skills, creativity, independence, and leadership qualities of the students they produce are important. These student characteristics are a measure of the faculty’s success in producing excellent researchers, and are addressed in this survey instrument.

            Respondent Information:

            Name of faculty member completing questionnaire: ______________________________________________________________________________
            Name of student being evaluated ____________________________________________________________________________________________
            Name of program with which student affiliated: __________________________________________________________________________________ ______________________________________________________________________________________________________________________
            Please use a separate form for each student who graduated and rate him or her on a scale of 1 = excellent, 3 = satisfactory, to 5 = very poor.

            How would you rate this student with regard to the following characteristics: Scale (circle one) Excellent Very Poor
            1. Graduate coursework relative to other students? 1 2 3 4 5
            2. Commitment, drive, determination, and perseverance? 1 2 3 4 5
            3. Creativity and imagination, in terms of experimental 1 2 3 4 5 interpretation as well as design?
            4. Technical abilities? 1 2 3 4 5
            5. Keeping up with and understanding the literature? 1 2 3 4 5
            6. Output and effectively completing tasks (e.g., 1 2 3 4 5 translating observations into a presentable paper)?
            7. Ability to write clearly and persuasively? 1 2 3 4 5
            8. Ability to speak clearly and persuasively? 1 2 3 4 5
            9. Leadership qualities (in the lab and/or program)? 1 2 3 4 5
            10. Being well organized (good record keeping and 1 2 3 4 5 well prepared lab notebooks) and effective time management?
            11. Potential for independence (i.e., are they on a 1 2 3 4 5 successful career trajectory)?
            12. Overall assessment as a productive scientist? 1 2 3 4 5

            26 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Existing Sources of Data for Graduate Program Self-Assessment

            Pending conduct of the surveys, programs may wish to turn to existing sources of data, both internal and external, to benchmark their success in achieving educational and programmatic goals. The following tables present various characteristics that programs may wish to examine relative to their faculty, graduate student populations, and career outcomes of recent graduates.

            Table 1

            ASSESSMENT OF FACULTY AND PROGRAM CHARACTERISTICS

            Data Internal Sources of Data Sources of Comparison Data Publication rates of faculty Faculty evaluations PubMed/MEDLINE Citation statistics regarding Faculty evaluations ISI Citation Index, faculty research NRC

            Research Doctorate

            Programs in the United States: Continuity and Change

            Grant portfolio of faculty Sponsored projects office NIH Web Site, WebCASPAR Cooperative research with Sponsored projects office industry Awards and honors to Department office Honorific societies and program faculty academies Faculty/student ratios Program office Peterson’s

            Graduate Programs in the Biological Sciences

            Demographic characteristics Program office of faculty Publication rates of faculty Faculty evaluations PubMed/MEDLINE Types and amount of training Program office NIH, NSF support (NRSA, TAs, RAs, etc.) Accreditation Program office Peterson’s

            Graduate Programs in the Biological Sciences

            Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 27

            Table 2

            ASSESSMENT OF GRADUATE STUDENT POPULATIONS

            Data Internal Sources of Data Sources of Comparison Data GRE scores Dean of Graduate School Educational Testing Service Department Office Undergraduate GPAs Student applications Peterson’s

            Graduate Program in the Biological Sciences

            Honorary scholarships Fiscal officer, department Proportion of underrepresented Student applications NRC Research-Doctorate

            Research-Doctorate

            minorities and women

            Programs in the United States: Continuity and Change

            Length of prior research Student surveys Not available experience Time to degree Registrar’s office NRC

            Research-Doctorate Programs in the United States: Continuity and Change

            Retention/Attrition Department Peterson’s

            Graduate Programs in the Biological Sciences

            First year GPAs Registrar’s office Graduate GPAs Registrar’s office “Admission to Candidacy” Program office standing Awards, fellowships, and Program office NRC

            honors received by students

            Share of training grants from Registrar’s office external scholarshipsNRC

            Continuity and Change

            Student publication and Program office citation rates

            Number of degrees conferred Program officer NRC

            28 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Table 3

            ASSESSMENT OF RECENT GRADUATES

            Data Internal Sources of Data Sources of Comparison Data Post-degree employment Development office Commission on Professionals and placement statistics Program office in Science and Engineering Survey of Doctorate Recipients Amount of external research Federal agencies support Involvement of graduates in Federal agencies, academies, panels national committees, panels, policy bodies Receipt of special honors by Not available graduates Publication and citation ISI Citation Index statistics of graduates Career status well beyond Development office, Survey of Doctorate graduation (e.g., 5 or 10 graduate survey Recipients years out)

            Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 29

            29

            Other Resources

            The literature and the Internet contain a number of resources that graduate programs may find useful as they embark on the task of assessing their own success. Some of the more notable works are cited below:

            ARTICLES

            Holzemer, W.L. “Doctoral Education in Nursing: An Assessment of Quality, 1979-1984.”

            Nursing Research. Vol. 36, No. 2. March/April 1987.

            Vol. 36, No. 2. March/April 1987.

            This article demonstrates how changes in the quality of graduate education in a particular discipline (in this case, nursing) can be assessed through questionnaires that capture data on quality-related program characteristics. Eighteen doctoral programs in nursing participated in 1979 and again in 1984, providing data on faculty training and accomplishments, student ability and performance, resources, academic and social environments, program processes, and alumni achievements. Readers may cull from this paper ideas about quantifiable characteristics of their own programs to assess.

            Kassebaum, D.G. “The Measurement of Outcomes in the Assessment of Educational Program Effectiveness.” Academic Medicine. Vol. 65, No. 5. May 1990.

            Academic Medicine. Vol. 65, No. 5. May 1990.

            Although the context of this paper is medical education, rather than Ph.D. training, a number of parallels can be drawn with regard to assessment activities. The author explains why outcome measures have become an area of increasing emphasis in educational assessment, illustrates how they have been used, and provides a model linking various educational goals to quantifiable indicators.

            BOOKS AND REPORTS

            Association of American Universities Graduate Education Committee.

            Graduate Education

            Report. Association of American Universities: Washington, D.C. 1998.

            This report, which describes the state of graduate education at AAU member institutions, includes a chapter on “best practices,” which is relevant to the task of self-assessment and program improvement. The chapter includes recommendations relevant to recruitment, admissions, graduate curriculum, mentoring, program evaluation, and other topics.

            Commission on Professionals in Science and Technology.

            Road Map for Conducting

            Employment Surveys of Doctoral Graduates in S&E. CPST: Washington, D.C. 1998.

            . CPST: Washington, D.C. 1998.

            This publication is intended to assist science and engineering professional societies and others in conducting surveys concerning the employment of recent doctoral graduates. It discusses survey design, data collection and management, and reporting. It may be of use to graduate programs that wish to conduct examinations of the career outcomes of their own graduates. 30 Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000

            Association of American Medical Colleges Council of Graduate Schools Task Force on Academic Review of Graduate Programs. Academic Review of Graduate Programs. Council of Graduate Schools: Washington, D.C. 1990.

            This guide takes the reader through the process of program review. It is applicable to graduate programs in all disciplines and offers much useful and practical advice.

            National Research Council.

            . National Academy Press: Washington, D.C. 1995.

            This very thick volume provides a wealth of data on virtually all research doctorate programs in the country. Representative data elements include demographic characteristic of students, average time to degree, number of degrees conferred, citation statistics of program faculty, and so on. The 1995 report includes data collected in 1993.

            National Research Council. Summary Report 1996:

            Doctorate Recipients from United States Universities. National Academy Press: Washington, D.C. 1996.

            Data on recipients of research doctorates awarded by U.S. universities from July 1, 1995 through June 30, 1996 may be found in this report, which derives from the 1996 Survey of Earned Doctorates. It contains dozens of tables with statistics on the demographics, disciplines of study, sources of support and other characteristics of new doctorates.

            Peterson’s Graduate Programs in the Biological Sciences. Peterson’s: Princeton, New Jersey. 1998.

            This publication, largely oriented to students exploring graduate school possibilities, provides a narrative description of each program that includes a few useful statistics as well, such as the numbers and demographic characteristics of faculty and students, average age of students, numbers of applicants and matriculants, minimum entrance exam requirements, and so forth.

            University of California Council of Graduate Deans.

            Excellence at Risk: The Future of

            Graduate Academic Education in the University of California. University of California: Oakland, California. 1997.

            The subject of this document is how the University of California system should build and sustain the highest quality graduate programs in the face of the many challenges confronting them. The recommendations may be applicable to other systems and include increasing academic specialization; rigorously measuring quality, productivity, and student outcomes; collecting and using data to inform decision-making; and attaining student body diversity.

            Association of American Medical Colleges

            Self-Assessment of Graduate Programs in the Biomedical Sciences — 2000 31

            31

            INTERNET SITES

            Association of Graduate Schools (AGS)

            The AGS is a body of the Association of American Universities (AAU) that brings together its graduate school deans. Its Web site includes data from a recent survey conducted to assess the steps that its membership has taken to modify graduate programs, including such issues as timeto- degree, program size, interdisciplinary programs, and career advising and placement. Access:

            http://www-ags.ucsd.edu/ags.html

            NIH Grants Page

            Data on individual grantees and principal investigators can be obtained at this site, and basic reports produced. Access:

            http://grants.nih.gov/grants/award/award.htm

            NSF Division of Science Resources

            This site allows access to tremendous amounts of data, though prepared data briefs and Web CASPAR, from the Survey of Earned Doctorates, the Survey of Doctorate Recipients, and the Survey of Graduate Students and Postdoctorate Fellows in Science and Engineering. Also, federal grant data by institutions can be obtained and multiple cross-tabulations performed. Access:

            http://www.nsf.gov/sbe/srs/stats.htm

            Peterson’s Education Center, Graduate and Professional Study

            This page offers information on graduate programs across the U.S. as well as background information on graduate education issues. Access:

            http://www.petersons.com/graduate

            PhDs.Org

            This site, which has a tremendous amount of information and links related to employment and careers for Ph.D.s, has a page specifically dedicated to ranking programs. The unique feature of this page is that it allows users to rank various characteristics of programs according to their importance, and then, drawing from data contained in the NRC publication

            Research-Doctorate

            Programs in the United States: Continuity and Change, the page orders programs according to the degree to which they fit the user’s stated priorities. Access: http://www.phds.org/ratings