A whole lot of Doctors are in the building today!

Last week was a very eventful and exciting week here in CARS. We had 4 dissertation defenses in a row. We’d like to congratulate Megan Rodgers Good, Jerusha Gerstner, Matthew Swain, and Rory Lazowski; who are now all able to put the letters Dr. before their names! Below are some pictures celebrating this special occasion.

Jerusha Gerstner presenting during her dissertation defense on Friday, April 3rd.

Jerusha Gerstner presenting during her dissertation defense on Friday, April 3rd.

Dr. Jerusha Gerstner and her dissertation committee. From left to right: Dr. Jeanne Horst, Dr. Monica Erbacher, Dr. Chris Coleman, Dr. Debbi Bandalos (Chair), and Dr. Dena Pastor

Dr. Jerusha Gerstner and her dissertation committee. From left to right: Dr. Jeanne Horst, Dr. Monica Erbacher, Dr. Chris Coleman, Dr. Debbi Bandalos (Chair), and Dr. Dena Pastor

Megan Rodgers Good presenting during her dissertation defense on Wednesday, April 1st.

Megan Rodgers Good presenting during her dissertation defense on Wednesday, April 1st.

Dr. Lazowski and his committee. From left to right: Dr. Robin Anderson, Dr. Dena Pastor (Chair), Dr. Debbi Bandalos, (on the screen) Dr. Chris Hulleman

Dr. Lazowski and his committee. From left to right: Dr. Robin Anderson, Dr. Dena Pastor (Chair), Dr. Debbi Bandalos, (on the screen) Dr. Chris Hulleman

March 2015 Progress Report

CARS

Progress toward achieving the CARS mission during the month of March was evidenced by the following activities and achievements:
PRESENTATIONS:

Schubert, C., Baller, S., Ott, Walter, K., Sapp, L., Swanson, M., & Jacovidis, J. (2015, March). Developing a health sciences information literacy assessment for undergraduates. Poster presented at the 2015 Association of College and Research Libraries Conference, Portland, OR.

PUBLICATIONS:

Horst, S.J., Ghant, W.A., & Whetstone, D.H. (2015). Enhancing assessment through the use of mixed methods. Assessment Update, 27, 4-14. Doi: 10.1002/au

Pastor, D.A. & Finney, S.J. (2015). Quantitative psychology. In J.G. Irons & R.L Miller (Eds.) Academic Advising: A handbook for advisors and students volume 2: A guide to the sub-disciplines. Retrieved from the Society for the Teaching of Psychology website: http://teachpsych.org/ebooks/academinc-advising-2015-vol2

Sessoms, J. & Finney, S.J. (in Press). Measuring and modeling change in examinee effort on low-stakes tests across testing occasions. International Journal of Testing.

CARS ASSESSMENT NEWS and SERVICE:

  • SASS has been reviewing the website constructed by the Student Affairs Assessment Council. They have discussed avenues which through which SASS may be more integrated on this website as a resource for student affairs practitioners. 1st year Doctoral student Kristen Smith has been critical in summarizing this feedback. Feedback to the SAAC website has also been submitted.
  • John Hathcoat and 1st year doctoral student Courtney Sanders developed a SACSCOC workshop proposal. This proposal aims to demystify the concepts of reliability and validity in the instrument selection/design process. This workshop may be critical for not only assessment at the program level but also for selecting performance indicators of institutional effectiveness.
  • 1st year doctoral student Kristen Smith and Tia Mann from the Substance Abuse Prevention program at JMU had a presentation accepted for the annual conference of the Association for the Assessment of Learning in Higher Education that will take place in June 2015. This presentation trains assessment practitioners on implementation fidelity.
  • The results from the STEM Competition have been disseminated for 2015. This year’s competition ended in a tie between Physics and Biology. Congrats to the winning seniors.
  • John Hathcoat and members of the Cluster 2 committee are in the process of developing training materials for establishing criteria to evaluate hand written responses to video. They are in the process of pursuing funding for summer training and coding. As of now, a Q-sort has been constructed for each prompt. Faculty will be trained on how to use the Q-sort to generate ideas for scoring criteria for a rubric. Most of the training materials have been developed, though additional material will need to be created for the audio. Two teams of faculty will likely meet. The first team will use the Q-sort to provide insight into the initial stages of a rubric whereas a second team will review audio recordings of each focus group to inform possible scoring criteria. A plan has also been developed to ensure that Cluster 2 will demonstrate student learning outcomes by Spring 2020.
  • 3rd year Doctoral student Matt Swain traveled to Radford College on March 30th to present to their staff about Assessment Day practices at JMU. Matt served as a GA on the Assessment Day team for two of the three years of his Doctoral program, and in his second year led the Assessment Day GA team.
  • Quantitative Psychology MA student Liz Pyburn successfully defended her thesis titled “Persons can speak louder than variables: Person-centered analyses and the prediction of student success” on Friday, March 6th. Her committee consisted of her advisor, Jeanne Horst (Chair), Monica Erbacher, and Keston Fulcher. Congratulations to Liz!
  • Jeanne Horst and John Hathcoat nominated first year doctoral student Kristen Smith for the JMU Student Employee of the year. Kristen won runner up for her leadership of the Student Affairs Assessment Support Services. Thanks for your hard work, Kristen! Glad that you could be recognized for this.
  • Quantitative Psychology MA student Heather Harris successfully defended her thesis titled “Propensity score matching in higher education assessment” on Wednesday, March 25th. Her committee consisted of Jeanne Horst (Chair), Monica Erbacher, and Keston Fulcher. Congrats to Heather! What an accomplishment!
  • Cluster One representatives met with Dr. Carol Hurney, Executive Director of the Center for Faculty innovation, to discuss assessment report writing. The meeting focused on strategies for report writing that would facilitate meaningful consumption of assessment results. Some of the suggestions included sharing assessment instruments with affiliated programs, framing results by research question (rather than by objective), and discussing the results in light of the more meta-skills or attitudes addressed (rather than in light of specific item writing). Further, Dr. Hurney encouraged Cluster One to ‘tell stories’ with their data, rather than simply reporting it.
  • The APT online submission system is officially open for 2015 APT submissions. All assessment coordinators for academic degree and certificate programs received email notification of this on March 10th.
  • On March 10th, CARS hosted an item writing workshop for the core section of the Information Literacy Test for seniors. Librarian Kathy Clarke led the session where 30 new items were reviewed and revised. There were four graduate students and four faculty members present to assist with the item critiques and revisions. Pizza was enjoyed by all!
  • Assessment Day makeup sessions were held throughout the month of March. There were multiple proctored sessions held in the Assessment & Testing Center for students who were unable to attend Assessment Day. 241 students were invited to attend one of the four scheduled sessions that took place in the evenings and were proctored by CARS GAs. Some of the sessions had to be rescheduled due to snow days, and the team finally wrapped up the last session as of March 20th. After the four sessions were held, any students remaining were directed to go to the Assessment & Testing Center to complete their assessment on a walk-in basis. These students also had holds placed on their accounts that would only be removed upon completion of the necessary assessments. We placed 119 holds, which is a greater number than usual, but we believe that this may have been influenced by the snow cancellations. Students are taking their assessments every day, and we continue to remove holds as they are completed.

PhD PROGRAM NEWS:

  • Laura Williams successfully defended her dissertation titled “The Effect of Examinee Motivation on Value-Added Estimates” on March 17th, 2015. Members of Laura’s committee included Donna Sundre (Chair), Monica Erbacher, and Jeanne Horst. Congratulations to Laura on this accomplishment; this makes her our 44th graduate of the Assessment & Measurement PhD program.
  • Although they were both accepted to other programs, we are thrilled to announce that both Heather Harris and Liz Pyburn have selected to stay at JMU to complete their PhD in our Assessment & Measurement program. Heather & Liz will be graduating from the Psychological Sciences MA program with a concentration in Quantitative Psychology in May. Both Heather and Liz have served external GA positions doing assessment work in the offices of Residence Life and Career and Academic Planning while they were enrolled in the MA program. We are so glad they chose our program to continue on for their PhD and we look forward to having them in CARS for another 3 years!
  • CARS continued their search for a new faculty colleague. We brought in three wonderful candidates for three day campus visits filled with interviews and presentations. We were very pleased with the quality of the applicant pool, and our expectations were met with the candidates. An offer has been extended and accepted and we are very excited to bring Ms. Allison Ames, who will be graduating from the University of North Carolina – Greensboro to JMU to join our faculty. We are very grateful to a hardworking, dedicated search committee co-chaired by Debbi Bandalos and John Hathcoat. Deborah Kipps-Vaughn and David Chase also served on the search committee.
  • 3rd year doctoral student Megan Rodgers Good has accepted a position as the Director of Academic Assessment at Auburn University. She will be the chief advisor to the Provost on all matters to assessment. Megan graduates in May and will begin her new position at Auburn on July 13th. Congratulations to Megan on this new and exciting opportunity. We look forward to seeing where her career takes her!

CARS VISITORS:

  •  Emily Hoole from the Center for Creative Leadership (CCL) in Greensboro, North Carolina traveled to JMU to deliver a CARS Talk titled “Assessment for Development: Assessment and Measurement Applications in the Leadership Development Field.” Dr. Hoole, a graduate of JMU’s Assessment & Measurement PhD program provided a very stimulating talk pertaining to the CCL, the Assessment for Development field, and new frontiers in assessment of leadership effectiveness. Emily, who serves as Group Director of Global Research and Evaluation is scheduled to being a three-year stint in Singapore in summer 2015.
  • On March 11th, CARS served as host to a team of five visitors from Osaka University. The team was comprised of Tatsuo Kawashima, Professor; Yukiki Abe, Associate Professor; Kazuya Yamaguchi, Professor; Hiroki Minamioka; and Junko Seno, Administrative Staff. The morning was spent with two General Education cluster coordinators, the Associate Vice Provost, two cluster assessment liaisons, and the Director and Associate Director of CARS. Morning discussions focused on JMU’s general education history, philosophy and framework. The afternoon meeting focused on assessment practice at JMU with Keston Fulcher, Donna Sundre, and Dena Pastor serving as facilitators. It was a very full day. We also found time to share some southern foods at lunch, provide a tour, and shop at the JMU bookstore.

Upcoming Dissertation Defense Announcements

With the arrival of spring also comes a season of dissertation defenses for our graduating PhD students. Below is a schedule of defenses that will be held in the coming days/weeks. Dissertation defenses for doctoral candidates are open to the public.

Wednesday, April 1st, 2015 

Megan Rodgers Good will be defending her dissertation titled “Improving Student Learning in Higher Education: A Mixed Methods Study” from 1:00 – 2:00 P.M.

Thursday, April 2nd, 2015

Matthew Swain will be defending his dissertation titled “The Effects of a Planned Missingness Design on Examinee Motivation and Psychometric Quality” from 3:00 – 4:00 P.M.

Friday, April 3rd, 2015 

Jerusha Gerstner will be defending her dissertation titled “Addressing Serial-Order and Negative-Keying Effects: A Mixed-Methods Study” from 10:00 – 11:00 A.M.

Rory A. Lazowski will be defending his dissertation titled “A Meta-Analytic Tutorial and a Narrative Review on Motivation Interventions in Education” from 1:00 – 3:00 P.M.

Thursday, April 9th, 2015 

Mandalyn Swanson will be defending her dissertation titled “Extending an IRT Mixture Model to detect Random Responders on Non-Cognitive Polytomously Scored Assessments” from 3:00 – 4:00 P.M.

All defense presentations will take place in Anthony-Seeger Hall, Room 9. The presentations will be 45 minutes long, with a 15 minute question and answer session. Committees will then deliberate and discuss with the candidates; this portion of the defense is closed to the public.

We hope you can come and join us in celebrating the culmination of hard-work and research these students have accomplished over their three years in our doctoral program.

Dissertation Defense for Laura Williams

We are pleased to announce that third-year doctoral student Laura Williams successfully defended her dissertation on Tuesday, March 17th, 2015. Laura’s dissertation was titled “”The Effect of Examinee motivation on Value-Added Estimates”. Laura’s committee was very pleased with her presentation and her work. Her committee members were Dr. Donna Sundre (chair), Dr. Jeanne Horst, and Dr. Monica Erbacher. Congratulations to Laura! We wish her continued luck while she searches for jobs in the area. Laura will graduate as the 44th graduate of the Assessment & Measurement PhD program.

Dr. Laura Williams and her advisor & committee chair, Dr. Donna Sundre.

Dr. Laura Williams and her advisor & committee chair, Dr. Donna Sundre.

Winter in CARS

Snowy campus

It sure has been a crazy winter here in Harrisonburg. We have had many snow days, as have most of the schools in the area; but that somehow hasn’t stopped us from being productive. It has been a very fruitful winter here in the Center. Here are a few notes to highlight what our faculty and students have been up to.

  • We have been honored with hosting multiple groups of scholars from Japan; who have all come to the United States, and to JMU to look for best practices in assessment. Our first visitor, Dr. Satoko Fukahori, from the National Institute for Educational Policy Research of Japan, was able to join us for our Spring Assessment Day, where she witnessed a large part of our assessment model in action! Earlier this month Donna Sundre met with a visiting group from Osaka University. The group was very eager to learn about assessment practices at JMU, particularly regarding General Education. The CARS and University Programs folks worked together to set up a day of meetings that we hope met all of their expectations. They were also excited to enjoy some local “southern” style foods! We are so humbled by the outreach from our colleagues from Japan, and look forward to further expanding the global culture of assessment that already exists today.
  • A faculty search is currently underway here in CARS as we look for someone to fill Dr. Keston Fulcher’s position as Keston prepares to move into the Executive Director role effective July 1st. As we mentioned previously, Dr. Donna Sundre will be retiring at the conclusion of the spring semester, as she sees her last two doctoral advisees through their dissertation defenses and graduation. Keston, as you may know, has been our Associate Director for some time, as well as having a dual role like all our faculty split between teaching and assessment. At this time we are looking to fill Keston’s position in regards to the assessment/teaching role, and will select a new Associate Director at a later date. The search has been underway since late fall and we are hopeful to be able to complete the process and hire someone very soon.
  • Keston Fulcher, Laura Williams (3rd yr. Doc), and the graduate assistants in the PASS office are gearing up for a new year of APT submissions. This past year brought a new option for a program to select to complete an alternate project in place of their typical APT. Programs would have to meet special criteria to be offered this alternative, such as receiving a score of exemplar for two reporting years in a row. The idea is that the programs will take this alternative year to explore new ideas, or perhaps work on an assessment related project they have been unable to accomplish as a result of having to keep up with the APT reporting cycle. As we go into our second year offering this option, we are pleased to see that programs are taking advantage of this opportunity. We can’t wait to see how these programs will use their time. Additionally, a change to element 6A on the APT will be implemented this year, so Keston and his GA’s have been hard at work preparing for that, meeting with programs to introduce the changes. Stay tuned for more on that in our monthly progress reports!
  • CARS has been working hard to prepare for a move out of Anthony-Seeger scheduled for June. For those alumni who are reading, you may be sad to know that when you return to visit, we will no longer be in the same place geographically that you once called home for 2-3 years. As with most changes, we are sad to leave the comfort of our space, but are eager to see what the new building has to offer. We will be moving into the newly named “Lakeview Hall” where we will, in fact, have a view of the lake. A spectacular one at that! This move is an opportunity for the center to get some new furniture and spruce up our space to become a true showpiece of an office. Keston Fulcher, David Yang, Paula Love, and Sharon Sipe have been working with the university’s property coordinators to ensure that the move is as seamless as possible. We will be sure to share an update with pictures and our new address when we get into our new space in June. A perfect time of year to come visit us on the lake and feed the ducks! (Don’t quote me on that, I don’t know if we are allowed to feed them.)

While this isn’t even close to a full list of the things that have been happening here, it’s a little glimpse into what our winter has been like! We are excited that spring has arrived, as we all prepare to head off to Chicago for the AERA/NCME conferences, and the ongoing dissertation and thesis defenses of our soon to be graduates! Keep checking back for announcements as our students finish up their time here at JMU.

James Madison University wins CHEA Award

We are excited to announce that James Madison University was named the 2015 winner of the Council for Higher Education Accreditation (CHEA) Award for Outstanding Institutional Practice in Student Learning Outcomes.

From the CHEA News Release:

“The 2015 CHEA Award recognizes the university for assessing the quality of its assessment practice, a form of meta-assessment. This effort has involved review and rating of the institution’s assessment practices across academic programs, monitoring assessment quality, addressing strengths and weakness and using the assessment system to communicate how assessment has affected student learning.”

Dr. Donna Sundre from the Center for Assessment & Research Studies, as well as Dr. Jerry Benson, Provost of Academic Affairs at James Madison University will travel to the CHEA annual conference in Washington D.C. on January 26th to receive the award.

CHEA Award Sundre

Congratulations to all who are involved in assessment at JMU. This award is truly the result of a collaborative effort among hard working assessment practitioners across the university. Thank you for your contributions to the culture of assessment at JMU.

You can read the full press release here.

December 2014 Progress Report

CARS

Progress toward achieving the CARS mission during the month of December was evidenced by the following activities and achievements:

Presentations:

Amato, H., Good, M. R., Smith, K.L., & Fulcher, K.H. (2014, December). Developing an Assessment Culture: Strategic Approaches to Faculty Development in Assessment. Workshop presented at the annual meeting of the Southern Association of Colleges and Schools; Commission on Colleges (SACSCOC) in Nashville, TN.

Fulcher, K.H., Good, M.R., & Coleman, C. (2014, December). Evaluating the Quality of Academic Degree Program Assessment. Workshop presented at the annual meeting of the Southern Association of Colleges and Schools; Commission on Colleges (SACSCOC) in Nashville, TN.

Fulcher, K.H., Rodgers-Good, M., Smith, K.L., & Coleman, C. (2014, December). Evidence of Improved Student Learning: Unpacking Comprehensive Standard 3.3.1.1 Presentation at the annual meeting of the Southern Association of Colleges and Schools; Commission on Colleges (SACSCOC) in Nashville, TN.

Publications:

Fisher, R., Smith, K., Finney, S. & Pinder, K. (2014). The importance of implementation fidelity data for evaluating program effectiveness. About Campus, 19, 28-32.

Fulcher, K.H., Good, M.R., Coleman, C.M., & Smith, K.L. (2014). A simple model for learning improvement: Weigh pig, feed pig, weigh pig. (Occasional Paper No. 23). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

CARS Assessment News and Service:

  • Sara Finney was appointed to the editorial board of the journal Educational Assessment. Educational Assessment publishes original research on the design, analysis and use of assessment to deepen understanding of the performance and Quality of individuals, groups, and programs in educational settings. It welcomes rigorous theoretical pieces and empirical studies (quantitative and qualitative) that can inform important national and international discussions in educational policy and practice. Sara also serves on the editorial boards for the following journals: Educational & Psychological Measurement, International Journal of Testing, and Journal of Educational Psychology.
  • On Dec 2nd, Sara Finney & Dena Pastor were guest lecturers for PSYC 200 (Orientation to Psychology). Dena and Sara presented their annual talk on “What is Quantitative Psychology?” and fielded questions about this area of psychology (training, jobs, etc.). We hope this introduction to the field will prompt students to pursue this area of study and possibly our graduate programs.
  • The Cluster 2 committee is making preparations for a series of focus groups to be conducted during the spring 2015 Assessment Day. The group is currently selecting segments of a foreign film to present to the focus group as well as designing questions for discussion aligned with their objectives.
  • John Hathcoat and Scott Paulson (Cluster 3 Coordinator) are working to finalize a second draft of a “Highlight Page” summarizing assessment results to be sent to all faculty who teach Cluster 3 courses. This is a two-page document presenting results for entering freshmen as well as the most recent sophomore-junior post-test analyses.
  • SASS held their final meeting of 2014 at Beyond on December 16th to celebrate their accomplishments this semester. Within the month of December, they have developed a draft of the SASS website.
  • Jeanne Horst and Ph.D. student Courtney Sanders attended the Diversity Council Meeting. Members of SASS, along with Quant. Psych. MA student Thai Ong, will be working with members of the Diversity Council on revising the MUDAS diversity scale.
  • John Hathcoat and Ph.D. student Courtney Sanders gave presentations to the Student Affairs Assessment Advisory Council in the month of December. Courtney presented on creating and mapping curriculum, while John presented on writing objectives.
  • Last month we announced that the search for a new Executive Director of CARS had concluded, and that Dr. Keston Fulcher would be assuming the position after Donna’s retirement in July 2015. We have now begun the search for a new faculty member to take over Keston’s teaching and assessment responsibilities. A search committee has been formed, comprised of faculty within CARS, outside of CARS, and also graduate student representation. The job description has been approved by Human Resources and the vacancy will be posted to the Chronicle of Higher Education, as well as InsideHigherEd.com and various electronic listservs. Committee review of applications will begin on February 1, 2015.
  • Jeanne Horst had an article accepted for publication. This is work she completed with one of our Student Affairs clients, Walt Ghant, from Community Service Learning; and Psych Sciences M.A. graduate Devon Whetsone. Congratulations to Jeanne, Walt, and Devon on this accomplishment!
  • A. student Thai Ong, and first-year Ph.D. student Kristen Smith submitted a proposal for a presentation at the JMU Diversity Conference. Thai has been collaborating with the SASS team to better understand how to define and measure diversity awareness at JMU. The proposal highlights assessment work from the Alternative Break Program. Good luck to Kristen and Thai!
  • Keston Fulcher was recognized by the Daily News Record as one of the top 10 business leaders under 40. (Article found here) Congratulations, Keston!
  • Sara Finney joined the technical advisory committee (TAC) for the Educational Research Bureau (ERB) in NYC. ERB is a global not-for-profit organization providing admission and achievement assessment as well as instructional services for PreK-Grade 12. Sara joins the following colleagues on this TAC: Joan Herman (UCLA-CRESST), Kurt Geisinger (UNL – Buros Center for Testing), Steve Sireci (UMASS – Center for Educational Assessment), Susan Brookhart (Educational Consultant), Vincent Alfonso (Gonzaga University), Laine Bradshaw (University of Georgia), and Thanos Patelis (National Center for Assessment). Sara began her three-year term at the November TAC meeting and will attend meetings twice yearly.
  • On December 11th, Sara Finney, Carolyn Miesen, and Matt Swain attended the stakeholders meeting for Transfer Orientation. This meeting focused on articulating important outcomes that will shape new programming and new assessment associated with Transfer Orientation. That is, stakeholders have determined that outcomes previously established (and used to guide programming and assessment) are no longer critical. Thus, efforts are being put forth to articulate the important learning and development outcomes.
  • On December 13th, John Sessoms, a student in the Quantitative Psychology concentration of the Psychological Sciences MA program, graduated with his Master’s degree. John was our only December graduate this year, and a reception was held in his honor on Friday, December 12th. Congratulations, John!
  • On December 12th, Sara Finney, John Hathcoat, Courtney Sanders, and Heather Harris attended the Student Affairs Assessment Advisory Council meeting. This meeting focused on 1) examining and giving feedback on the SASS website (supported by CARS), which John presented; and 2) the importance of mapping programming to objectives, which Courtney presented. Prior to the meeting, Sara noted the chairs of the council need to include an Associate Vice President on the council (much like the Academic Affairs AAC has an Associate Vice-Provost as Chair and member). The issue of committee membership may be a topic of discussion during the spring semester meeting.
  • Quantitative Psychology Master’s student Kate Pinder successfully defended her master’s thesis proposal on December 11th. The title of her proposal is “The Effects of Ordinal Data on Coefficient Alpha.” Committee members included Debbi Bandalos (chair), Dena Pastor, and Monica Erbacher. Congratulations to Kate!
  • Courtney Sanders and Jeanne Horst met with the Diversity Council Assessment subcommittee on December 10th.
  • Donna Sundre attended the final meeting of the JMU SACSCOC Advisory Council. This was an important meeting, as many members had just returned from the annual regional accreditation meeting. JMU is ahead of the curve with new assessment, accreditation, and accountability mandates. This Council’s mission is to assure that JMU conducts its institutional effectiveness research with our own needs and concerns assuming primary status. We need to prepare for change in higher education.

Assessment & Measurement Ph.D. News and Service:

  • SACSCOC staff confirmed that they will support a two month internship for A&M doctoral student, Kristen Smith, this summer. As part of her duties, she will help the accrediting body evaluate the overall effectiveness of the QEP requirement. Kristen and Keston will work with SACSCOC this spring to develop the methodology.
  • Doctoral student Rory Lazowski successfully defended his dissertation proposal on December 16th. The title of his proposal is “Motivation Interventions in Education: A Systematic Review.” Committee members included Dena Pastor (chair), Debbi Bandalos, Robin Anderson, and Chris Hulleman. Congrats, Rory!
Follow

Get every new post delivered to your Inbox.