Study Abroad screen shot

Technology: There Are No Silver Bullets

Print Friendly

Summary

IUPUI uses the Open Source Portfolio tools in Sakai, customized to the needs and preferences of faculty, students, and staff of the Indiana University system. Because it is embedded in our LMS, the environment is used for a range of teaching, learning, and assessment purposes. Despite its advantages, the platform looks and feels dated. Its complexity is both beneficial and daunting, and users’ expectations today are colored more by Facebook and Twitter than by the first LMS platforms. Thus, IU is exploring future technology options, and ePortfolio users at IUPUI are providing leadership in articulating requirements.

Authors

Susan Kahn
Director, ePortfolio Initiative and Director, Office of Institutional Effectiveness

Susan Scott
ePortfolio Coordinator and Assistant Director, Institutional Effectiveness

Overview: IUPUI’s Technology Story

Part I: Our Technology Story

Platform

IUPUI currently uses the Open Source Portfolio (OSP) tools in Sakai. The IU system-wide implementation of Sakai is known as Oncourse and, in many respects, is customized to the needs and preferences of IU students, faculty, and staff members. The portfolio software environment, embedded in Oncourse, can be used for a wide range of teaching, learning, and assessment purposes. Two or three IUPUI programs have chosen alternate portfolio platforms that they believe better meet their needs and those of their students, though one of those is converting to the current system.

Using a portfolio platform integrated with the university’s LMS offers a number of academic benefits. For example, when students submit course assignments for review and/or grading, an assignment-linking feature can simultaneously place the material into a portfolio. Advisors can obtain access to their advisees’ portfolios in combination with their course planners through the official university records system. Using embedded rubrics, faculty can aggregate rubric scores for artifacts or whole sections of ePortfolios and generate reports that greatly facilitate program-level assessment. For students, the ability to limit access can help them feel comfortable about testing new ideas and sharing personal opinions with others within the closed community of a course, program, or other internal group. Another advantage, though  not unique to Oncourse, is that the portfolio tools allow a student easily to create multiple portfolios for different audiences or purposes.

The institutional ability to add and customize functionality to meet local users’ needs has been an advantage of our eportfolio platform, but has resulted in an abundance of features that can be daunting. For example, to create forms necessary to build portfolio templates, users must manipulate SML files. Most faculty thus need help from technicalportfolio matrix from English capstone staff to create a basic portfolio template. Users have commented that the LMS, including the portfolio tools, requires too many clicks to accomplish basic tasks. Embedding media files can be difficult. Since the portfolios are in the LMS, a secure system, some students who wish to share their ePortfolios beyond the campus community find that process difficult. University Information Technology Services (UITS) allows students access to their portfolios for five years after the most recent enrollment, and students can download or save their portfolios for longer-term use; however, many students find it complicated to edit and update their portfolios outside the LMS. A frequent complaint among both faculty and students is that the platform is not “intuitive” or is otherwise “difficult to learn.”

Historical Overview

Our current platform, Sakai, was selected by UITS, which at the time was seeking open-source solutions to learning management and other functions. The Sakai Project ultimately merged with the Open Source Portfolio Initiative. As far as we know, there was no particular selection process focused on ePortfolio tools, and IUPUI constituents had little opportunity for input.

IUPUI’s initial ePortfolio pilot implementation in 2004-05 revealed numerous flaws in the platform. We considered moving the campus to a different platform, but UITS committed to work closely with us to make needed improvements. In collaboration with campus academic units using the ePortfolio, we drew up specifications for software development in 2007, and UITS dedicated developer time to improve performance, resolve problems, and add new features. IUPUI shared with UITS the cost of modifying the limited presentation functionality, and, by 2009, the platform (or at least the IU instance) had dramatically improved.

Oncourse is now a decade old, however, and it looks and feels dated. In Fall 2012, UITS engaged the university system in an initiative to explore next-generation learning technology tools, including the ePortfolio platform, providing us a welcome opportunity to seek a more intuitive, engaging ePortfolio that responds to student interests as well as to faculty needs. As those with the most hands-on experience, IUPUI ePortfolio users are leading the committees charged with identifying possible new platforms. A Joint Working Group on ePortfolio Platform Review, with leadership from IUPUI, took advantage of the AAEEBL/EPAC webinar series of platform demonstrations to identify functional requirements for a new system and to survey the ePortfolio market. UITS has committed to consider Working Group recommendations and to a multi-year transition process for any new platform selected.

Support and Collaboration

University Information Technology Services (UITS) assumes responsibility for platform licensing, hosting, supporting, and adjusting features for all campuses in the IU system. Units that prefer to use another product do so at their own expense. Though few have chosen to do this, most of those have opted for free (or baseline free) services and expect students to find other sources of technical help they may occasionally need. UITS has a formal intercampus advisory committee to provide feedback about directions for Oncourse, with an ePortfolio Advisory Committee providing additional recommendations. The large staff of UITS includes a signficant cadre focused on maintaining relationships with faculty on all campuses, and the instructional technology staff in the campus centers for teaching and learning are funded by UITS. These consultants provide an important source of feedback to UITS regarding concerns and needs for new development.

Though UITS has its own strategic vision for technology at Indiana University, it takes its system-level mission seriously and seeks to accommodate the different campus cultures, teaching and research emphases, and administrative requirements at the same time that it challenges all to consider benefits available with emerging technologies. Many of the senior UITS managers are faculty or teach as adjuncts in schools from business to liberal arts to education. Though the Vice President for Information Technology reports to the university president and is a member of the system-wide President’s Cabinet, campus-level reporting is typically to the academic vice chancellor on each campus.

Part II: Pedagogy Drives Technology

chart of eportfolio adoption over timeThe earliest ePortfolio adopters at IUPUI were primarily interested in program assessment, especially for specialized accreditation. Since the basic software in those early projects included only a rudimentary presentation capability, that assessment bias was perhaps not surprising. As more prospective users expressed interest in web presentation, the IUPUI ePortfolio team worked with UITS to upgrade that functionality, and adoption of the ePortfolio has correspondingly accelerated, with more projects focused primarily on fostering academic engagement and supporting particular pedagogies. (The graphic illustrates the rapid acceleration of adoption following implementation of improved presentations.)

At the same time, other improvements have been made. Today’s environment supports not only assessment and reporting, but also scaffolding of student portfolio work, peer and instructor feedback, reflection, integrative learning, learning enhancement, and showcasing of student growth over time. An unexpected number of graduate and professional programs have adopted or expressed interest in adopting the ePortfolio.

portfolio screen shotMost programs that have traditionally used paper or digital but not web-based portfolios (such as journalism, interior design, and performance arts) continue to do so, at least for the present. One such program, philanthropic studies, is currently beginning the transition, and the Herron School of Art and Design is considering adoption. The electronic Personal Development Plan (ePDP), however, demonstrates a key value added by a web-based format: portability across courses and terms as well as greater student investment in documenting learning. The PDP was originally developed as a paper binder. One of the reasons for pursuing an alternate strategy was that students often perceived the plan as a series of discrete exercises, rather than a unified, coherent document, and were thus not particularly invested in the resulting plan. Paper PDPs were often left in faculty offices at the end of the semester and never picked up, and faculty struggled with the sheer bulk of documents (and boxes of binders clogging their offices). Students’ response to the electronic version has been generally positive–more enthusiasm, more identification with the resulting portfolio, more thoughtfulness about whether their academic and career goals mesh with their interests and strengths, as well as evidence that at least some students are indeed continuing to develop their ePDPs in subsequent semesters.

Evidence

We cannot attribute improvements in pedagogy to platform alone; many of the observed improvements in teaching and learning could likely have developed in any portfolio platform, because they resulted from curricular improvements necessitated by the shift to electronic format. Often, the initial planning process for portfolio adoption led to improvements when, for example, discussions about curriculum-mapping revealed gaps. (We sometimes refer to this phenomenon with the phrase “moving from my course to our curriculum.”) Review of student portfolios has in several cases led to modification of course or program content to increase attention to particular competencies or to reduce perceived duplication. This is consistent with the way in which a majority of IUPUI schools understand assessment and pedagogy to be entwined.

As noted in the Engaging Students section below, we have surveyed students who have just begun using ePortfolios for the ePDP, and their reactions to the platform have been primarily positive or neutral, with first-year students more likely to experience problems. We have also surveyed and interviewed faculty involved with ePDP implementation; a slight majority have anticipated or experienced challenges attributed to the platform, especially those who engaged in little professional development or training before starting. See our Catalyst Professional Development Story for information about how faculty helped us improve platform deficiencies following the initial campus-level rollout a decade ago.

Connections to Other Sectors of the Catalyst

Pedagogy

One feature of Oncourse frequently used with the ePortfolio is the ability for students to secure formative feedback from other students and from instructors for individual artifacts or cells or the whole matrix and then revise as needed before submission for a final summative evaluation. This functionality permits more extended and private feedback than the limited “Comment” feature of the presentation portfolio tool. See our Social Pedagogy and Reflection practices for examples of use of this capability in group contexts. Used well, the platform can support faculty pedagogical choices as well as do other platforms (more, indeed, than any one group of faculty may want to pursue). It accommodates High Impact Practices well, though social media must be linked rather than embedded and video files such as for digital storytelling are best handled via YouTube links. The protections of student information that concern faculty and administrators (if not students themselves) make our platform’s security features important. At the same time, that security requires extra steps for students to share their ePortfolios publicly, even though open access may be desirable for the pedagogical purposes of reaching authentic audiences.

Professional Development

The Oncourse portfolio platform is challenging for most faculty, who usually require the assistance of a trained instructional technology consultant (with appropriate authorizations to work in the environment) to set up a matrix in the portfolio site. Though determining how to structure the matrix is mainly a matter of logic and disciplinary thinking, this too sometimes requires the support of instructional designers. With complexity and flexibility come choices to make, a situation that adds to the perceived difficulty of the system.

An instructional design consultant typically works with an individual instructor or group of faculty to think through course and program learning outcomes and teaching strategies appropriate for the purpose at hand, but these matters of instructional and program design would arise with any portfolio platform. Faculty who have used Oncourse regularly in their courses can usually make the knowledge transfer to create their own assignments, incorporate appropriate rubrics, and use the reporting functionality to generate reports in real time if needed. website screen shotSuch needs can also be addressed by simple video tutorials such as those provided on the University College ePDP web site. Those less confident may require multiple consultations, and the heavy time commitment of consulting staff represents one drawback of using such a complex, customizable platform.

Most of our professional development is now focused on pedagogy, though we do provide one or two hands-on workshops each year to help potential adopters get a feel for the portfolio environment. (See our Professional Development Story for further information.) Through our grant program from 2006-2011, we grew to understand the extent to which we were asking faculty to transform their practice. In almost every case, faculty begin with the idea of learning how to use a technology tool to accomplish their goal. Through consultations, symposia, and support of travel to present at inter/national conferences, we worked to help them move beyond focusing on the technology to emphasize curriculum, design, pedagogy, and assessment principles. What we learned from grantees and continue to learn from workshop attendees is fed back into developing new topics for our workshops and symposia, including sessions on curriculum mapping, rubric development and use, reflection, ePortfolio pedagogy, assessment with ePortfolios, and web design principles.

UITS supports Oncourse and the other basic platforms provided university-wide, including the ePortfolio. Most technology support takes place in individual or group consultation with the Center for Teaching and Learning, which designates a consultant to function as the resident ePortfolio expert. Another CTL staff member provides instructional design consultation for ePortfolio users as time permits. website screen shotThe online UITS Knowledge Base includes numerous quick-start guides, detailed instructions for a wide range of common actions in Oncourse, and tutorials on many of the more difficult tasks. One unit of UITS, IT Training, conducts group training workshops in person and via webinar recordings, several of which now cover ePortfolio basics for both faculty and students. We encourage faculty to take responsibility for introducing their students to the ePortfolio, but help is available in special cases. For the ePDP–now encompassing over fifty sections of the First-Year Seminar each fall, with more to be added in future, and with new faculty each semester–IT Training staff will visit classes by arrangement for orientations to the platform.

Outcomes Assessment

Our ePortfolio platform functions very effectively for many of the assessment processes at IUPUI. As described above, assessment was an early driver for adoption of ePortfolio; identification of platform improvements was driven in large part by programs intending to use the ePortfolio for assessment and specialized accreditation. The most commonly used feature for all types of assessment is the matrix, a table wherein the individual cells can be structured to juxtapose a variety of elements–for example, horizontal rows may reflect program learning outcomes while vertical columns represent courses, and students upload one or more assignments (perhaps a research report and a reflection on the research project) into cells to document their accomplishment of the designated outcomes. Faculty, field experience supervisors, or others with whom the student has shared the portfolio can then assess the combined documents in one or several cells to determine that the outcome has been met. The portfolio reporting function enables faculty and program leaders to generate reports in real time about the status and/or level of completion across a course, activity, or degree program at several levels from global to individual artifact. Report data can also be exported for manipulation in complex analysis software. When the matrix is linked to the presentation function, the uploaded artifacts also become available to use as attachments or pages of the web-based portfolio.

Scaling Up

As explained in our Scaling Up Story, platform weaknesses seriously damaged the initial ePortfolio rollout at IUPUI. Originally enthusiastic faculty became detractors after experience with the early iteration of our software, and bad memories linger. Nonetheless, we worked carefully both with faculty experimenters and UITS software developers to identify, prioritize, and address critical problems. Once we were able to offer an improved technology platform to students and faculty, scaling up ePortfolio has been more positive.

Engaging Students

Faculty and advisors in portfolio sites have access to students’ work, including their portfolios, but otherwise students must take action to share their ePortfolios with other students or people outside the university community. Among the many ePortfolio projects at IUPUI, some (though not all) are intended to encourage student ownership of their learning; portfolios created in capstone courses typically emphasize integrative learning and metacognition. One of the main reasons for digitizing the electronic Personal Development Plan (ePDP), the focus of our C2L project, was to support not only the literal ownership of a presentation-style portfolio, but more importantly to foster students’ sense of agency, self-authorship, and ownership of learning from the start of their undergraduate experience.

UITS is responsible for providing technical support for student use of the various technology systems, including the portfolio, so students can call/email/chat with Help Desk staff 24×7 to resolve specific operating problems. The IT Training unit within UITS offers in-class orientation sessions as well as online introductory tutorials. The ePDP web site includes a section of online tutorials and guides for students.

More general support and encouragement is sometimes offered by a sponsoring academic unit. For example, the First-Year Seminars in which students prepare their initial ePDP are taught by an instructional team including instructor, advisor, librarian, and peer mentor. The student mentors have taken on the responsibility to advocate for and model portfolio preparation and to help students get started. In addition, IUPUI’s campus-wide service learning program is support by service learning assistants, scholarship students who create ePortfolios as part of their own training and development and who can thus assist faculty and students with service learning projportfolio screen shotects using ePortfolios. The Life-Health Sciences Internship Program employs two experienced interns as “student ambassadors,” whose responsibilities now include supporting students in preparing their internship portfolios. Our Center for Research and Learning piloted a similar arrangement for the Summer 2013 undergraduate research program.

The campus has not systematically surveyed all students who have prepared portfolios about their opinions, in part because comparison across the widely varied project structures would be difficult. Course evaluations for First-Year Seminars include questions about the ePortfolio platform, and only 20 percent to 25 percent of these students report problems using the technology (or, to frame the results more positively, 75 to 80 percent report no particular probems). It might be noted that these new students are also learning how to use Oncourse, which may compound their perception of difficulty. Capstone students already familiar with Oncourse seem to find the ePortfolio less difficult. Two graduate programs using ePortfolios surveyed their students, who reported no noteworthy problems with the technology. Anecdotally, we have noticed that students are more likely to report difficulty when faculty begin by apologizing for how difficult or awkward the technology is or when faculty themselves appear to struggle with or avoid using the technology.

Attachments and Supporting Documents

IU 2013 ePortfolio Requirements Building

Conclusion: Looking Forward

UITS is currently engaging campus constituents in the search for the next generation of learning technologies, with several LMS pilots in 2013. An IU-wide committee on ePortfolio Platform Review has worked to identify and prioritize ePortfolio technology needs and to identify candidates for close review. As the earliest and largest adopter of ePortfolios, IUPUI is heavily represented on this body, which is co-chaired by one of our C2L project directors. The group produced a detailed list of requirements (attached below), ranging from basic functionalities for document management to those that support social and reflection-based pedagogies and those that enable assessment management. Improved user experience and ability to integrate with other enterprise systems are also high priorities.

The committee also reviewed the ePortfolio platform demonstrations archived on the AAEEBL web site. One of these vendors provided a live demonstration, which proved very beneficial; we learned that the technology would not meet our needs for either assessment or presentation, something that was not evident in the recorded demo. We issued a Request for Information in Summer 2013 and have carefully examined the lengthy, helpful responses during Fall 2013. We have scheduled live demos of those platforms that appear to have the greatest potential to meet our needs.

In addition to looking at functional needs to support learning and assessment, the committee has been charged to consider such issues as:

  • How will each prospective platform serve IU’s needs? How is the platform unique from others in this respect?
  • How critical is the platform for meeting the university’s needs?
  • Will the platform meet IU standards for security and data storage?
  • How much and what kinds of support will the platform require?
  • Can IU create a working relationship or even a partnership with the vendor?

It is also fair to say that it is difficult to make informed judgments about potential platforms without knowing what LMS will be selected. For example, many Learning Management Systems now include sophisticated assessment capabilities, which might then not need to be duplicated by the ePortfolio platform.

 

Leave a Reply

Your email address will not be published. Required fields are marked *