What We’ve Learned

Print Friendly

Sometimes what you have learned and what your data say are not the same.  When faced with this, one must examine both sets of data.  Anecdotal evidence from both student and faculty users of ePortfolio are positive; however, our survey data do not fully reflect the anecdotal experiences we have heard from faculty and students.  We plan to conduct more in-depth qualitative case studies where we can carefully examine the variables affecting faculty and student experiences and learning outcomes.

Author: Alison S. Carson

Evidence

Collecting evidence regarding the effectiveness of ePortfolio implementation on our campus has been essential for two audiences: 1) current and future granting agencies require evidence, and 2) our own campus administration also wanted evidence of the effectiveness of our project.  Tightly controlled studies examining the isolated effect of the implementation of ePortfolio on student learning are hard to do. In absence of this approach, we have worked to examine the implementation of ePortfolio in the following ways: 1) comparing final course grades, DFWI rates, and retention rates in ePortfolio and non-ePortfolio sections in our First-Year Program (instructor could not be controlled for), and 2) comparing NSSE items in rates in ePortfolio and non-ePortfolio sections.

Working with the Registrar’s Office was key as we depended on that office to first denote courses that included ePortfolio in the course, and then gather the needed data to do the evaluation.  Such dependence also meant that getting access to the data was, at times, challenging.  As such, our evaluation data is not as up to date as we would like it to be.

With regard to student success, we began by comparing eP sections in the FYP with non-eP sections in the FYP for both the pilot in Spring 2011 and the first semester of implementation in Fall 2011.  For Spring 2011, when comparing final cumulative GPAs, the average final cumulative GPA for non-ePortfolio sections was 3.019.  The average final cumulative GPA for only those students in the ePortfolio sections of FYP in Spring 2011 was 3.1375.  This is a slight increase in cumulative GPA for those students in the ePortfolio course.  While these numbers were encouraging, more recent numbers have shown more sporadic findings. For Spring 2012 classes, no significant differences were found comparing course GPAs in ePortfolio and non-ePortfolio sections (MnoneP=2.85; MeP=2.87).  (See Table 1 for additional findings.)

Table 1. Comparison of Course GPA in ePortfolio and non-ePortfolio FYP classes.

Term

eP Course GPA

Non-eP Course GPA

Fall 2011

3.19

3.16

Spring 2012

2.87

2.85

Fall 2012

3.06

3.11

Spring 2013

3.20

3.09

 

In addition to course outcome information (some of which we have been able to gather), we have also used evidence from the Connect to Learning Core Survey.  This survey collects information on student and faculty perceptions of and experiences with ePortfolio.  Additionally, it uses items from the NSSE survey (National Survey of Student Engagement).  We also give this survey on our campus biannually, and therefore we are able to compare the national findings to both our campus at large and our ePortfolio courses, more specifically. 

In Fall 2012, we had 41 sections of classes (502 students) using ePortfolio.  In Spring 2013, we had 60 sections (675 students) using ePortfolio.  However, our response rate to our core Survey was considerably smaller.  In Fall 2012, 162 students from 14 different class sections responded to the survey (32% and 34% respectively).   In Spring 2013, 96 students from 9 different class sections responded (14.2% and 15% respectively).

A review of the Core Survey data is interesting, although deeper analysis of the data across the campuses, as demonstrated by Helen Chen, is likely to be more revealing than a more cursory overview of the data on a single campus.  Our Core Survey data reveal that students are not as engaged with ePortfolio as we had hoped (see Table 2).  For example, only 44% of students stated that they strongly agreed or agreed with the statement “I hope my other professors decide to use ePortfolio in their classes.”  Although almost 60% of students strongly agreed or agreed with the statement “I would recommend to other students to use ePortfolio.” While this is an improvement, we are hopeful that these numbers will increase.

While our course GPA data did not capture the effectiveness of ePortfolio, this simply means that ePortfolio may not be effective in this particular way.  Student engagement is another measure we examined, and here we have more positive results (see Table 3).

Table 2. Percent of sample responding to the Core Survey items with “Strongly Agree” or “Agree.”

Core Survey Question

% 3’s&4’s

 

Building my ePortfolio helped me to think more deeply about the content of this course.

55.39%

My peers/classmates provided useful feedback on my ePortfolio.

45.32%

Using ePortfolio has allowed me to be more aware of my growth and development as a learner.

46.67%

Someday, I’d like to use my ePortfolio to show what I’ve learned and what I can do to others, such as potential employers or professors at another college.

56.88%

My instructor provided useful feedback on my ePortfolio.

84.82%

Building my ePortfolio helped me to make connections between ideas.

58.85%

 I know that my instructor looked at my ePortfolio.

95.63%

My instructor(s) discussed the ways ePortfolio helps students to learn.

83.41%

I know my  peers/classmates looked at my ePortfolio.

52.83%

Someday, I’d like to show my ePortfolio to my friends and family.

42.22%

Building my ePortfolio helped me focus on planning for my education.

41.26%

I enjoyed building my ePortfolio.

46.69%

The ePortfolio was an important part of this course.

83.48%

Building my ePortfolio helped me succeed as a student.

49.55%

When I needed it, help building my ePortfolio was available.

85.71%

Doing the ePortfolio helped me learn about myself.

50.60%

I feel that I was more stimulated and engaged with the course content because of the use of ePortfolio.

55.11%

I feel that the use of ePortfolio in the course helped me to better examine my process of writing.

55.80%

I feel that the use of ePortfolio in the course helped me to better examine my process of learning.

54.66%

I feel that the use of ePortfolio in the course has allowed me to be more creative.

56.69%

Through the use of ePortfolio, I feel that I have been able to develop an academic identity.

43.30%

The use of ePortfolio in this course has helped me improve my technical / computer skills.

62.33%

I would recommend to other students to use ePortfolio.

59.45%

I hope my other professors decide to use ePortfolio in their classes.

44.64%

 

Table 3. Items from the NSSE comparing National, Manhattanville and Manhattanville ePortfolio course responses.

To what extent has your experience in this course….. % responding Quite a bit or Very much Portfolio sections Non-ePortfolio sections Comparable NSSE data
Emphasized synthesizing and organizing ideas, information, or experiences in new ways? 91.7% 75.7% 64.1%
Emphasized applying theories or concepts to practical problems or in new situations? 88.7% 83.8% 63.5%
Emphasized knowledge, skills, and personal development in writing clearly and effectively? 84.2% 43.2% 65.5%
Contributed to their knowledge, skills, and personal development in understanding oneself? 78.19% 91.9% 58.5%
Contributed to their knowledge, skills, and personal development in working effectively with others? 83.4% 75.7% 64.7%

 

In addition to these student data, we have also been collecting data from our faculty. From a review of the data from the faculty information sheets, faculty are requiring the use of ePortfolio, grading the work in the ePortfolio and using ePortfolio in a deeply integrated way in their courses.  Most faculty were expecting students to update their ePortfolios at least once a week, and giving feedback to the students between 2-4 times a month.  This suggests that despite the deep use of ePortfolio in the classes by the instructors, the students are not fully understanding or feeling the educational benefits in the classroom, as measured by items in the Core Survey.

We have also gathered data regarding the participation in professional development around the use of eP. We have worked hard to develop an inclusive professional development experience, engaging full and part-time faculty and staff.  Since we began our professional development TLC series in Fall 2010, we have included 118 faculty and staff.  In this group, 74 full-time faculty have participated.  This accounts for 71.8% of our full-time faculty.  Of that 76, 41 (54%) have implemented ePortfolio into a class. Five out of 15 part-time faculty (33%) are using ePortfolio.  A number of staff have also participated, examining ways to use ePortfolio in their offices.  Twenty-one staff members from campus offices such as Center for Career Development, the Library, School of Education offices, and Academic Advising have taken ePortfolio training. Of those staff members, 9 (42.8%) now use ePortfolio either in their office or in a class.

Following each of our Teaching and Learning Circles, we ask participants to complete a brief survey.  This survey provides us with feedback about the utility and perception of the professional development as well as providing us with some understanding of what faculty and staff are getting out of the TLC.  To this end, perhaps most useful are the open ended questions.  In response to the question “What was the most useful thing you learned during the TLC?,” one participant wrote: “E-port is not only NOT a warehouse…it is a tool with endless possibilities for extending learning into cognitive domains not possible with conventional instruction.” Another wrote: “That I can be in touch with partners and students in a realistic way. Also, I can learn from the others. Creativity is also special in this kind of communication. It is not cold.” One hundred percent of the Fall 2012 TLC participants who responded to the survey stated that they planned to use ePortfolio in a class. Of the 12 TLC participants, 6 were faculty and 3 have implemented ePortfolio into a class. One is currently leading an ePortfolio TLC this Fall 2013.

We have also been working to grow our use of ePortfolio in the First-Year Program.  All Academic Writing instructors are using ePortfolio, to varying degrees.  In addition, we have increased our use of ePortfolio to 43.75% of seminar sections.  Clearly, this is an area in which we could improve, but we are making slow and steady progress.

Beyond the individual numbers, the re-emergence of faculty development on our campus has had a significant effect.  We feel strongly that the benefits of the professional development, which was the first on our campus after a long period of no professional development, had much to do with the institutional support for a Center for Teaching and Learning, which came to fruition in the Spring of 2012.  As a result of our TLCs, the conversation around teaching and learning is changing.  It has been a culture shift, which has only increased in speed with Sherie McClam as the founding director of the CTL.

 

Overall Findings and Implications

Our overall data suggest that more sophisticated data collection and data analysis are needed.  Course GPA data is influenced by so many factors, as is retention. Additionally, the varied use of ePortfolio by faculty, the various strengths and weaknesses of our students, motivation levels, technological proficiencies, etc., are likely to affect the ways in which students engage with and experience the use of ePortfolio in the classroom.  One approach that we plan to use to more deeply examine the student experience with ePortfolio is the case study.  By randomly selecting a sample of ePortfolios from different courses, we can engage in an indepth analysis of the ePortfolio content as well as interview the ePortfolio creator to better understand the experience.  This is not an alternative approach to the use of broad strokes analysis such as surveys, but it will allow us to drill down and better understand individual experiences.

Project Dashboard

Faculty from various departments on our campus use ePortfolio:

  • Art History
  • Communications
  • Economics
  • Education
  • First-Year Program
  • Music
  • Physics
  • Psychology
  • Sociology
  • Studio Art
  • Women’s Studies

Over the course of the last two years of ePortfolio implementation, we have experienced the following growth:

Semester School of Education (SOE) Courses First-Year Program (FYP) Courses Arts and Science Courses
Fall 2011 7
Spring 2012 40 17 9
Fall 2012 20 28 12
Spring 2013 14 28 16
Fall 2013 17 18

 

We are experiencing steady growth in the implementation of ePortfolio in our First-Year program as well as our Arts and Sciences courses.  In the FYP, all Academic Writing instructors are using ePortfolio, to varying degrees.  In addition, we have increased our use of ePortfolio to 43.75% of sections of seminar instructors.  The use of ePortfolio in School of Education courses has been more variable.  ePortfolio was getting heavy use from SOE faculty as New York State now mandates that all student teacher candidates submit an ePortfolio.  NYS then selected Pearson as the ePortfolio vendor to be used.  Therefore, many SOE folks have chosen not to deal with two ePortfolio systems, even though the two products are vastly different. Digication has since become a recognized platform and has developed an EdTPA template for NY education teacher candidates to use.  This is only just been finished, so it remains to be seen how it will be used on our campus.

While the growth in our non-FYP, non-SOE courses has been slower than we would like, we are encouraged by the continuing increase.  We believe our strong professional development programming will support growth in this area and is responsible for this growth. We have worked hard to develop an inclusive professional development experience, engaging full and part-time faculty and staff.  Since we began our professional development TLC series in Fall 2010, we have included 118 faculty and staff.  In this group, 74 full-time faculty have participated.  This accounts for 71.8% of our full-time faculty.  Of that 76, 41 (54%) have implemented ePortfolio into a class. Five out of 15 part-time faculty (33%) are using ePortfolio.  A number of staff have also participated, examining ways to use ePortfolio in their offices.  Twenty-one staff members from campus offices such as Center for Career Development, the Library, School of Education offices, and Academic Advising have attended a TLC. Of those staff members, 9 (42.8%) now use ePortfolio either in their office or in a class.

Over the course of the last two years, the growth in use of ePortfolio in courses reflects the steady increase of students in courses creating ePortfolios:

Semester School of Education (SOE) Students First Year Program (FYP) Students Arts and Science Students
Spring 2012 147 107 75
Fall 2012 26 359 117
Spring 2013 87 403 185
Fall 2013 94 329

 

In addition to the various courses in which ePortfolio is used, all students are given ePortfolio accounts upon enrollment at Manhattanville.  Additionally, all sophomores and seniors are required to submit their Manhattanville Portfolios, which use ePortfolio. Following three small pilot programs, Spring 2013 marked the official transition with all sophomores submitting their Portfolios via ePortfolio.  We experienced a much higher compliance rate of students submitting their Portfolios on time.  In past (paper) submissions, while specific data has not been collected, approximately 1/3 of students were submitting their paper Portfolios prior to or on the due date.  This Spring, we had 70% of students submit their electronic Portfolios prior to or on the due date.  We see this as a success!  We have long hoped that the electronic platform would increase student engagement with the Portfolio System.  While compliance may be a crude measure of engagement, it is one interpretation.  There are still kinks to be worked out in this system, including the electronic approval of the Portfolios by all faculty advisors, and the evaluation of the Portfolios.  Faculty response to the use of ePortfolio for the Manhattanville Portfolio System has been mixed.  Despite multiple opportunities for training, many faculty did not take advantage of the offers and experienced frustration when having to approve student submissions.  Additionally, the use of the technology led students to behave in unanticipated ways, which increased faculty frustration.  Specifically, with a paper Portfolio, students would come to see their faculty advisors in person requesting Portfolio approval, which provided an opportunity for discussion and revision.  With the ePortfolio, students skipped this step, simply requesting electronic approval via email.  Faculty emailboxes were flooded with emails from students requesting review and approval.  This amounts to a change in faculty advisor procedure requiring Portfolios to be submitted to advisors early for review.

ePortfolio is used in other areas of campus where faculty also engage in their use, including our Tenure and Promotions committee, and general education assessment.

We would not be where we are today without the support of our eTerns. We currently have approximately 10 eTerns.  We began our student ePortfolio support program in Spring 2012.  We have one student who serves as an eTern, who works very closely with our Instructional Technologist, who is the staff supervisor of our eTern program.  The Instructional Technologist works in our Center for Teaching and Learning.  The Director of the Center for Teaching and Learning, Dr. Sherie McClam, is also a member of our ePortfolio Team on campus.

 

 

 

 

Skip to toolbar