Skip to content. | Skip to navigation

Advanced Search…
Personal tools
 
You are here: Home → KBRIN Cores → Evaluation Core → Methods and Data

Methods and Data

Formative Evaluation

The Formative Evaluation stage includes:
1) An evaluability assessment with the purpose to investigate the

  • program’s rationale and conceptualization (Wholey, 1987),
  • the assumptions and expectations based on which the program was designed and is functioning, to describe the program model (Rossi, 2004), the dynamics of this collaborative network; and,
  • to assess program’s evaluation capacity (ex, existing records, existing databases, data collection capacity, etc).

2) A process evaluation that informs what the program does, whether it is or not delivered as intended, and whether the intended target population is successfully reached and served. The process evaluation focuses on the program implementation, and includes an assessment of the program’s operations and of the program’s interaction with the intended target populations.

Summative Evaluation

The strategy that we are developing builds upon past evaluation work conducted by several INBRE faculty members and on the results of the evaluability assessment and of the process evaluation.
The intent is to document and integrate all existing evidence of student participation in the program and to collect evidence on the impact of their research experience on their educational/career choices or plans. We also seek to investigate the quality of students’ research experiences and the processes through which specific outcomes arise.

Research Design

This is a population study; all current (prospective approach) and former (retrospective approach) KY-INBRE students will be invited to participate in the study.

The target population includes:
A direct target population that includes:

  • the undergraduate students engaged in the biomedical research at the primarily undergraduate institutions supported by INBRE in Kentucky, and,
  • the undergraduate students from such institutions who participate in the 10-week summer research program offered at the major research institutions – the University of Louisville and the University of Kentucky. 

and, an indirect target population is represented by the students who work in the laboratories that were developed and are supported with INBRE funding, and/or work with a faculty member who has an INBRE funded project. This group may include both undergraduate and graduate students.

Data Collection

The research evaluation involves a baseline student self-assessment of competencies necessary for a successful career in research, and at least one follow-up self-assessment at the end of project involvement, depending on the length of time a student participates in the project.

Using an online survey interface, we collect data on the participation in the undergraduate research programs using

  • pre/post online surveys for the group that participates in the intensive 10-week program at one of the two leading universities;
  • online surveys at the beginning of the first semester and at the end of each semester during the period  students participate in the research program offered during the academic year.

We are currently developing secure REDCap databases to easily track the funded projects and the participants in all KBRIN activities. 

Data Items

We collect evidence on expected impacts on students’ personal, intellectual and professional development, including effects on career choice and pathway; it is expected that majority of students pursued a career in medicine or a science-related field.

Using the set of URSSA competencies and/or others supported and developed by the stakeholders, we are developing tiered self-assessment questionnaires to measure students’ confidence and understanding of selected research concepts in specific biomedical research fields.

Our starting point is a self-assessment instrument developed and validated by a group at the University of Colorado at Boulder, the Undergraduate Research Student Self-Assessment (URSSA) that measures student gains in several domains. For more information regarding the URSSA tool please visit: http://www.colorado.edu/eer/research/undergradtools.html

The following student characteristics will be collected from all students who have in any way taken advantage of the KY-INBRE program since its inception (2000-2010), at any of the participant universities:

  • Socio-demographic information – race/ethnicity, gender, urban/rural residence, first generation of college student, traditional/adult student, full-time vs. part-time, etc; current employment status, years on college,
  • Student activities (ex, number of hours of work in lab, number of papers/posters prepared); and
  • Student satisfaction with research program
  • Student outcomes: cognitive, attitudinal, and behavioral

Data Analyses

Our ultimate outcomes of interest are both at aggregate level, such as the changes in the number of students who pursue graduate biomedical and health education, and at individual level, such as the impact the program participation has on students’ research knowledge, career choices, and personal development.

The data at aggregate level will be analyzed using descriptive statistics and presented using tables and graphs.

For the individual level data, several techniques are considered:

  • traditional psychometric techniques to test the reliability and validity of measures - such as factor and reliability analysis to inform with regards to the reliability and content validity of the scales, along with the scale dimensionality,
  • the item response theory (IRT) technique may also be used during the trimming process of scale items;
  • correlations and regression models will estimate the criterion and the construct validity of the scales.
  • to validate our scales, along with the tools developed for each tier and for each biomedical research area, we must include data items known to be correlated with or to be predictive of the learning outcomes we are interested in. 

Further, traditional descriptive statistics and predictive models those are feasible with the data available (ex, OLS regression, structural equation and multilevel models, time series – student outcomes before and after INBRE, developing student trajectories using individual growth models) will be conducted. More specifically, data will be analyzed for differences between groups (ex, UofL vs. UK, males vs. females, etc) using analysis of variance and t-tests; for associations between variables, using correlation coefficients (ex, we will compute the correlation between the confidence and applied understanding measures); and, multiple regression analyses will be used to explore how development of specific skills are predictive of career choices/ decisions. Other research questions will be developed/explored as we advance in the process of evaluation.

References

  • Laursen, S., Hunter, A.-B., Seymour, E., Thiry, H. & Melton, G. (2010). Undergraduate research in the sciences: Engaging students in real science. San Francisco: Jossey Bass.
  • Rossi, P. H., Lipsey, M. W., Freeman, H. E.  (2004). Evaluation: A Systematic Approach.  Seventh Edition.  Thousand Oaks, CA: Sage.
  • Stufflebeam DL, Shinkfield AJ. (2007) Evaluation theory, models, and applications. San Francisco: Jossey-Bass. ISBN: 9780787977658
  • Wholey, J.S. (1987). Evaluability Assessment: Developing Program Theory. New Directions for Program Evaluation, no. 33 (p77-92) San Francisco: Jossey-Bass.
  • Centers for Disease Control and Prevention Framework for Program Evaluation in Public Health
  • William Trochim’s website: www.socialresearchmethods.net
Document Actions
Personal tools