- Overview of ePortfolio-related Outcomes Assessment at IUPUI
- Part I: Setting the Stage
- Part II: Developmental Story
- Part III: Conceptual Framework
- Catalyst and Connector
- Inquiry, Reflection, and Integration
- Connections to Other Sectors of the Catalyst
- Scaling Up
- Professional Development
- Attachments and Supporting Documents
Most, if not all, of the approximately 40 programs that have adopted ePortfolio at IUPUI are making some use of it for assessment, but there is no single, monolithic set of assessment practices (or assessment rubrics) used by all. In many cases, especially for disciplines subject to specialized accreditation, assessment has been the primary driver of ePortfolio adoption. In others, programs that are primarily implementing ePortfolio for developmental or showcase purposes also recognize its value as a source of authentic evidence of student achievement. While we do not have a campus-wide practice of using ePortfolio to assess general education outcomes, several programs are using it for this purpose, and ePortfolio is written into the plan for a new general education distribution program implemented in Fall 2013.
Our challenge, if anything, is to persuade some programs to look beyond assessment, to enrich their use of ePortfolio by recognizing and availing themselves of its potential to support student learning and development. In some quarters, the perception that ePortfolio is primarily a vehicle for assessment has hindered its adoption. The electronic Personal Development Plan, our focus in C2L, holds promise to help broaden faculty understanding of what ePortfolio can contribute to curriculum, pedagogy, advising, and student development.
ePortfolio Coordinator and Assistant Director, Office of Institutional Effectiveness
Director, ePortfolio Initiative, and Director, Office of Institutional Effectiveness
Overview of ePortfolio-related Outcomes Assessment at IUPUI
Part I: Setting the Stage
IUPUI is a large, public, urban, decentralized research university. Its Division of Planning and Institutional Improvement (PAII), headed by a senior executive reporting to the chancellor, provides broad leadership and specialized support to the campus community, but “assessment” is everyone’s business, and assessment of learning outcomes is foremost a faculty responsibility. Since the early 1990’s, PAII leaders have fostered an institutional culture that values evidence-based decision-making and have worked to ensure that information about outcomes—particularly learning outcomes—leads to continuous improvement of curriculum, instruction, and academic support and administrative services. (See attached graphic representation of the cycle at IUPUI: IUPUI Planning Structure, Banta.) Influential campus-level committees such as the Council on Retention and Graduation and the Program Review and Assessment Committee assure broad faculty and professional staff representation in planning, advocacy, monitoring, and implementing the work of assessment.
IUPUI’s general education approach has been based on essential skills and characteristics similar to the AAC&U Essential Learning Outcomes. Faculty assess student accomplishment of these outcomes in multiple ways including a complex multi-year system of faculty assessments in courses using a common rating schema with results automatically exported to an institutional research database. The Office of Institutional Research annually reports back at the school and/or department level, and faculty study the cumulative data for needed improvement. Though valuable, this campus-level process obviates the use of ePortfolios, which are typically adopted by units interested in more qualitative assessment. Every degree program has articulated its own student learning outcomes, as have departments of the Division of Student Affairs. Professional schools with specialized accrediting bodies have been leaders in undertaking regular, comprehensive outcomes assessment. Certainly, some departments and faculty perceive assessment to be administrative make-work, but IUPUI is large enough that there remain many who are concerned to improve curricula based on solid evidence, and their influence grows each year. To learn more, see the IUPUI self-study report prepared in 2012 for reaffirmation of our accreditation by the Higher Learning Commission. The annual campus assessment report provides a more condensed overview of outcomes assessment work across the institution.
We recognize that at some institutions, the phrase “outcomes assessment” refers primarily to measures of accomplishment such as persistence and graduation rates, which are generally tracked and reported by an institutional research office. Certainly these metrics are important at IUPUI as well, and our Office of Institutional Research (OIR) regularly tracks and reports this data to all interested stakeholders (OIR is part of the Planning and Institutional Improvement Division mentioned above). In addition, University College has its own research department that conducts and reports extensive research on these kinds of outcomes with particular attention to entering students. Our ePDP evaluation data reported elsewhere on the Catalyst site comes primarily from that source. Nevertheless, because of the strong and long-term emphasis on assessment of student learning outcomes in the culture of IUPUI, we focus in this story on the meaning that would resonate with most of our ePortfolio adopters.
Part II: Developmental Story
IUPUI began planning for ePortfolio adoption in 2001 with outcomes assessment as a primary purpose. Portfolio leaders hoped that ePortfolios would become a common method for assessing student achievement of the Principles of Undergraduate Learning (PULs), our campus-wide general education outcomes. At the same time, a good prospect seemed to be new themed learning communities where several PULs might be introduced as part of the two to four linked courses. Though intended well, the implementation was top-down and seen by many as an administrative imposition. Many faculty and academic staff were engaged in planning for a pilot in 2004-05, but when unrealistic plans for campus-wide adoption, coupled with immature software, brought the pilot to an unsuccessful conclusion, it became necessary to regroup.
An office and budget had been created in 2002, and, fortunately, these continued to exist, drawing on funding and staff lines contributed by Academic Affairs, Planning and Institutional Improvement, and University Information Technology Services. From the outset, the initiative has thus incorporated a close alignment among teaching and learning, assessment, professional development, and technology. As we began to grasp the pedagogical and curricular implications of ePortfolio adoption, we shifted to offering small grants to departments with self-identified needs and willingness to be patient. We placed a priority on uses for PUL assessment, but over the next several years more proposals came from graduate programs who wanted to improve their assessment for specialized accreditation goals. These were joined by a few brave co-curricular units (undergraduate research, service learning, and student affairs) that saw opportunities to deal with their unusual assessment challenges. In the meantime, a few individual faculty members picked up ePortfolio as a means to organize, engage, and assess student learning within single courses (300-level American Studies courses, English and Spanish capstones). Use of ePortfolios continued to be focused primarily on assessment of student learning outcomes.
Development of improved presentation capability in 2009-10 fueled increased interest in adopting ePortfolios in many academic units and growth of “mixed use” portfolios, with University College as a campus leader. At the same time, we began more purposefully to solicit experimentation among the various engaged practices that are a cornerstone of an IUPUI undergraduate education: research, international study, service learning, and experiential learning such as internships and practica. These cross-cutting practices also helped generate wider interest in ePortfolio use, very often with an assessment focus (e.g., how does undergraduate research foster development of critical thinking skills? how does service learning contribute to civic understandings and attitudes?). We also worked with University College to create or strengthen linkages among these co-curricular practices and the electronic Personal Development Plan (ePDP) as an emerging undergraduate ePortfolio.
By Spring 2013, of the nearly 40 active ePortfolio projects, from single-course to full-program to co-curricular, 30 were using the ePortfolio either for program assessment and accreditation or for assessment of student learning or both. Most of those also have other purposes (e.g., integrative learning, development, career preparation, etc.). Three programs (one doctoral, one master’s, and one baccalaureate) have already undergone successful specialized accreditation review using ePortfolios to document and assess student learning outcomes. Some projects take advantage of reporting functionality IU has added to our platform (currently Sakai/OSP). Reports can provide an overview of scores (on whole portfolios, parts of portfolios, or individual artifacts and reflections) and enable drilling down to levels of increasing detail about scores and rubric dimensions.
Such assessment will need to continue to be undertaken and understood program by program, however, and undergraduate programs will continue to take different approaches from those of graduate programs or student affairs programs. Some, perhaps many, programs will continue to find ePortfolios unappealing for their students or other purposes. Fortunately, there is no expectation of either uniformity in approach or universal adoption. Faculty have numerous opportunities to hear about ePortfolios through formal channels such as professional development workshops and communications, but the “snowball effect” and hallway conversations with colleagues will continue to serve us slowly but well.
Part III: Conceptual Framework
Catalyst and Connector
Because of the existing and fairly pervasive “culture of evidence” at IUPUI, the ePortfolio Initiative has been primarily a connector of existing or developing assessment practices, although in some cases, it is beginning to foster new approaches to outcomes assessment. Program-level student learning outcomes were already being aligned with PUL learning outcomes or with accrediting bodies’ competencies; ePortfolio offered a means of sorting out these complexities for assessment purposes.
In 2012 the state legislature enacted a 120-hour cap on baccalaureate completion requirements (with some exceptions granted). Programs are working to meet the spirit of this legislation by several means, including broadening approaches to evidence-based credit award. The Organizational Leadership and Supervision program is sharing its ePortfolio template and learning modules for Prior Learning Assessment with other departments to facilitate faculty determinations regarding PLA across the campus. The same 2012 legislature also mandated a statewide transferable general education distribution schema, and IUPUI’s planning document includes use of student ePDPs throughout the general education core courses to support student development. There is potential here to position ePortfolio as a faculty/student-friendly way to undertake assessment of intended learning outcomes, grounded in the kinds of authentic practices faculty value.
Inquiry, Reflection, and Integration
All assessment of student learning outcomes at IUPUI is conceptualized as “faculty-led inquiry into student learning,” not just the assessment undertaken with ePortfolios. Faculty (and other academic and professional staff) do reflect on assessment results, whether these data are provided by Institutional Research or result from their own surveys of students, internship supervisors, employers, and alumni, from ePortfolio assessment, or from more traditional assessment methods. A key value that ePortfolios add to common practice (whether for PUL assessment or as part of program review and self-study) is that they can support nuanced understandings of strengths and opportunities for improvement. More than thirty of these faculty and staff (not including ePortfolio team members) have published their work in disciplinary journals and/or presented at AAEEBL, the IUPUI Assessment Institute, AAC&U events, and disciplinary conferences as well as campus-wide symposia and workshops other than those sponsored by the ePortfolio initiative. (A sampling of publications from faculty other than the C2L leadership team members is referenced below.)
The diversity of ePortfolio use at IUPUI makes it difficult to collect formal data about impact of ePortfolio-related assessment. Our ePortfolio Initiative annual reports share particularly interesting stories from adopters, and “impact” has been valued in terms of breadth of adoption across schools, departments, and divisions. As noted above, faculty and staff have engaged in scholarship around results of their work with ePortfolios that reflect the Catalyst Inquiry-Reflection-Integration principles. Several include mention of their work with ePortfolios in their annual assessment reports and have presented their work at meetings of the Program Review and Assessment Committee.
Connections to Other Sectors of the Catalyst
The association between pedagogy and assessment at IUPUI is substantial. A key part of that association is the expectation in our “Culture of Evidence” that “closing the loop” is integral to the assessment cycle. That is, a major purpose of assessment is to identify opportunities to strengthen programs and thereby improve their effectiveness in fostering student learning. Electronic portfolios are certainly not the only way of accomplishing that purpose, but they remain one of the most powerful sources of information to help explain and understand assessment results. Even those who initially adopt ePortfolio simply to collect examples of authentic student work to demonstrate compliance with specialized accreditation standards (and there are very few so perfunctory) quickly recognize three central pedagogical processes that ePortfolios address or facilitate: reflection, integrative learning, and social pedagogies.
The faculty interviews conducted for our research associated with Cohort VI of the Inter/National Coalition for Electronic Portfolio Research revealed examples in almost every case of how the central importance of reflection generated and focused faculty attention on pedagogy, specifically on ways to help students learn how to reflect at appropriate intellectual levels. Attention to reflection often led to further interest in how social pedagogies such as peer feedback and engagement with other authentic audiences could assist such learning. Similarly, even if not a preliminary goal, interviewees reported early notice of ePortfolio’s exceptional ability to foster integrative learning within a course, across a program, and/or between and among academic, co-curricular, work, and life experiences.
As our Scaling Up Story explains, campus assessment leadership and academic leadership have cooperated (along with university and campus technology leadership) to fund, staff, and support the campus ePortfolio Initiative. Though the initiative resides in Academic Affairs, the ePortfolio director and coordinator both have split appointments with Planning and Institutional Improvement. That collaboration also means that “scaling up” expectations include expanded ePortfolio use for assessment as well as for teaching/learning and other strategic priorities. In fact, mentions of ePortfolio in IUPUI’s new strategic plan (“Our Commitment to Indiana and Beyond“) are focused on assessment and career planning rather than enhanced learning.
Our Professional Development Story describes the trajectory of our ePortfolio activities from introducing the platform as a tool, largely for those interested in assessment, responding to identified interests with occasional workshops on curriculum mapping and rubrics, addressing the growing interest in understanding and fostering student reflection, and currently targeting broader topics of ePortfolio assessment, pedagogy, and student support. University College has added its own workshops on reflection geared particularly, but not limited, to improving reflection in the ePDP.
The Center for Teaching and Learning (CTL) limits its assessment-related faculty development to periodic workshops on course assessment and online resources about crafting learning outcomes. The PAII Division does not itself offer professional development, but PAII leaders work with officers of the large Program Review and Assessment Committee (PRAC) to embed learning opportunities in meeting agendas so that PRAC members can serve as assessment resources for their units. For example, recently the CTL representative provided a mini-course on writing learning outcomes, followed by PRAC members from the Schools of Education, Science, and Nursing each describing examples of assessing their program-level learning outcomes. The representative from IUPUI’s campus in Columbus, Indiana, focused on the Division of Education’s use of ePortfolio in both assessing accreditation standards and assisting students in developing professional showcase ePortfolios. At other PRAC meetings, the ePortfolio Director has organized individual presentations by faculty engaged in using ePortfolios in different ways to support assessment, and she reports more broadly on the initiative annually. The two approaches–of CTL and of PAII through PRAC–have created an opening for the ePortfolio Initiative to offer opportunities for individuals and teams of faculty to learn more about assessment via the context of ePortfolios.
Our Sakai/OSP platform both enhances and discourages assessment work, though the overall balance is positive. The complexity of the platform, with its many options and customizations, can prove daunting to faculty, most of whom need the support of an instructional technology specialist to create the forms necessary to use the matrix tool. Those interested in program-level assessment generally have sufficient incentive to persist, and our early adopters have often expressed how helpful they (and their students) found the matrix in visually organizing the connections between course and program outcomes or program outcomes and accreditation standards. Given a preponderance of assessment interest, the UITS software developers could justify time invested in developing a range of some twenty kinds of reports to organize data resulting from assessment in the ePortfolio matrix. Assessment needs are further addressed by the flexibility to export these reports to Excel or SPSS for further manipulation and study. On the other hand, as the title of our Technology Story states, “There Are No Silver Bullets,” and those using the IUPUI ePortfolio for assessment have been forceful advocates for improvements as we explore adopting a new platform.
Attachments and Supporting Documents
Anton, M. “Second Language Program Assessment and Students’ Reflections on Learning Outcomes.” Presentation at American Council for the Teaching of Foreign Languages, Annual Meeting, Philadelphia, PA, November 2012.
Applegate, R., & Irwin, M. “Electronic Portfolios for Program-level Learning Outcomes Assessment.” Presentation at Association for Library and Information Science Educators, Annual Meeting, San Diego, CA, January 2011.
McGuire, L., Gentle-Genitty, C., & Galyean, E. (in press) “The ePortfolio: Product and Process in Assessing Competencies for Social Work Education.” Special Edition on Technology and Social Work Education, Journal of Baccalaureate Social Work.
Meek, J., Riner, M.E., Pesut, D., Runshe, D., & Allam, E. (2013) “A Pilot Study Evaluation of Student Reflective Thinking in a Doctorate of Nursing Practice Program,” Journal of Nursing Education and Practice, Vol. 3, No. 8.
Steinberg, K., & Norris, K. (2011) “Assessing Civic Mindedness.” Diversity and Democracy, Association of American Colleges and Universities, Fall 2011.
Because IUPUI takes outcomes assessment seriously and because so many of the ePortfolio projects in the pipeline include assessment as one of their purposes, we anticipate the linkage will continue to be strong. If, as proposed, the new general education program adopts the ePDP portfolio, growth in ePortfolio use will be accelerated. At the same time, increasing numbers of capstone courses are experimenting with some form of ePortfolio; since most undergraduate and some master’s programs incorporate capstone courses or experiences, they seem another likely focus for the spread of ePortfolios. We will also see a different form of “closing the loop” as students who created an ePDP in their first-year seminar reach the senior capstone stage. There are already some experiments under way to adapt the ePDP for use in capstones.
As noted previously, the institution incorporates multiple active forms of outcomes assessment at all levels of study and type of program. Assessment with ePortfolios currently accounts for slightly over ten percent of academic programs and a number of co-curricular programs (which are very important at IUPUI), so ePortfolios can become an increasingly large proportion of a very high amount of assessment activity. Since IUPUI’s approach to assessment already embodies a process very similar to the Inquiry-Reflection-Integration framework, albeit with a different vocabulary, ePortfolio adoption will continue to accommodate existing IUPUI assessment models. On the other hand, as the nuanced qualitative information available from ePortfolios becomes more widely recognized, it may be that ePortfolios can be positioned as an opportunity (or at least the lesser of several evils) for those faculty and departments uncomfortable with quantitative approaches to assessment.