Stage 4: About the Redesign Results

This page covers the outcomes of the course redesign, including:

• Whether the redesign solved the original problem
• Whether and how the students mastered the SLOs
• Whether the students in the redesigned course were more successful than students in previous courses
• Any unexpected results from the redesigned course
• Graphs, tables, and sample student work demonstrating the changes between the original and redesigned courses
• Student feedback, including challenges the students may have encountered in the redesign
• Lessons I learned and redesign tips, including obstacles I encountered in the redesign and strategies to increase student engagement
• My reflection on the experience of redesigning the course, participating in the CRT Institute, and developing this ePortfolio

Stage 4 was completed on June 15, 2018.

Overview

The original redesign has as its pre-design comparison a section of the same course which I taught fully-online in Fall 2016, SOC 368-41. The original intent of this redesign was to see how changes in the course structure and presentation of its materials would change the outcomes of a fully-online section. Both the Fall 2016 course (“pre-redesign course”) and the Spring 2018 online course (“redesigned course”) were fully-online, asynchronous courses.

Quite unintentionally, however, this redesign also provides information on the difference between fully-online and in-person classes. In setting up my Spring 2018 sections, I copied the same assignment structure from the redesigned online course (368-42) into a Canvas course shell for my in-person SOC 368-02 section. The only addition to the in-person class was in-class assignments to apply what they learned from the lecture to current events, and the only subtraction was removal of the acknowledgement that the course was not a criminal justice/policing course and the online readiness quiz, which was specific to online classes. (The in-person class was notified during the first day of class that the course was not a criminal-justice/policing course.)

Apart from these changes, the content and the environment were substantially the same across both classes. The lectures were the same lectures, as were the homework questions, quiz banks, and research assignments. As a result, it is possible to compare the outcomes from this in-person section of the same course to the fully-online section which I redesigned, to see the differences between fully-online and in-person class results.

Some of the charts below compare only the predesigned and redesigned fully-online sections. Where possible or appropriate, I have included information on the in-person section as well.

Numeric Distribution Results

The redesigned class had more As, Bs, and Fs, and one additional student-initiated W. It also had fewer Cs, Ds, and WUs.

Overall, the redesigned course had slightly better outcomes than the pre-redesigned course.

Grade Percentage Distribution Results

Positive outcomes: The redesigned course had a higher percentage of passing scores (45%) compared to the pre-redesigned course (40%).

The redesigned course had a lower percentage of WUs (25%) compared to the pre-redesigned course (37%).

The redesigned course had a lower percentage of Ds (6%) compared to the pre-redesigned course (9%).

Negative outcomes: The redesigned course had a higher percentage of Fs (16%) and Ws (9%) than the pre-redesigned course (9% and 6% respectively).

Passing/Non-Passing Distribution Results

The redesigned course had a 4% increase in passing scores compared to the pre-redesigned course. However, the outcome is still not optimal, as the majority of students did not pass either course.

So did the redesign fix the problem it was intended to fix (the high DFWI rate)? Unfortunately, the answer to this question is “No.”

However, the redesign did give us additional information about certain other issues which may help in subsequent redesigns of this type of course.

Hypotheses and Results

Hypotheses concerning post-census drop rates:

• Hypothesis 1: As enrollee awareness of the course focus increases, post-census drop rates should decrease.
• Hypothesis 2: As enrolleee awareness of the demands of an online course increases, post-census drop rates should decrease.
• Hypothesis 3: As support for writing assistance for enrollees increases, post-census drop rates should decrease.
• Hypothesis 5: As visual and organizational design simplicity increases, post-census drop rates should decrease.
• Hypothesis 7: As active commitment to dedicated time for the course increases, post-census drop rates should decrease.
• Hypothesis 9: As perceived and actual workloads decrease, post-census drop rates should decrease.
• Hypothesis 11: As avenues for personal connection with the professor increase, post-census drop rates should decrease.

These hypotheses were supported in terms of the predicted result. More students dropped before the census date than after in the redesigned class.

In the pre-redesigned course (Fall 2016), pre-census drops totaled 12 (of which 1 was a WU drop), and post-census drops also totaled 12 (all of which were WU drops).

In the in-person redesigned course (Spring 2018, SOC 368-02), pre-census drops totaled 4 (no WU drops), and post-census drops totaled 5 (all of which were WU drops).

In the online redesigned class (Spring 2018, SOC 368-42), pre-census drops totaled 16 (of which 3 were WU drops), and post-census drops totaled 11 (of which 8 were WU drops).

Since so many changes were made to the class in the redesign, it is difficult to determine which intervention or interventions caused the pre-census drop rate to increase and the post-census drop rate to decrease.

Hypotheses concerning D, F and I rates:

• Hypothesis 4: As support for writing assistance for enrollees increases, course D, F and I rates should decrease.
• Hypothesis 6: As visual and organizational design simplicity increases, post-census D, F, and I rates should decrease.
• Hypothesis 8: As active commitment to dedicated time for the course increases, course D, F, and I rates should decrease.
• Hypothesis 10: As perceived and actual workloads decrease, course D, F, and I rates should decrease.
• Hypothesis 12: As avenues for personal connection with the professor increase, course D, F, and I rates should decrease.

These hypotheses were not supported in terms of the predicted result. These rates actually increased, rather than decreasing.

In the pre-redesigned course (Fall 2016), D, F and I rates totaled 6 (17%).

In the in-person redesigned course (Spring 2018, SOC 368-02), D, F and I rates totaled 3 (8%).

In the online redesigned class (Spring 2018, SOC 368-42), D, F and I rates totaled 8 (25%).

Intervention Review

Seven interventions were attempted: (1) raising enrollee awareness of the course focus, (2) raising enrollee awareness of the demands of an online course, (3) increased writing support, (4) increased visual and organizational design simplicity, (5) active commitment to dedicated time in the course, (6) decrease in perceived and actual workloads, and (7) increased avenues for personal connection with the professor.

(1) Raising enrollee awareness of the course focus

All students in the course responded to the request that they acknowledge the course focus (sociological rather than policing/criminal justice) by January 29th. This request may have contributed to the increased drop rates before the census period. However, no pattern connecting the timing of the students’ response to this request and the students’ final grades in the course appeared in the data.

(2) Raising enrollee awareness of the demands of an online course

All students in the course completed the online course readiness quiz, and submitted a plan for dealing with the issues identified by the quiz, by February 1st. However, no pattern connecting either the timing of the students’ completion of this quiz or the detail level of the plan with the students’ final grades in the course appeared in the data.

(3) Increased writing support

This was not provided, because I forgot to put the information into the course, due to some other issues that came up during the weeks leading up to the start of the semester. Although students had access to the writer’s workshops, it does not appear that students used these resources effectively.

(4) Increased visual and organizational design simplicity

Some students were so used to Blackboard that a shift to a new LMS actually created more confusion for them, although the visual design of Canvas is simpler than Blackboard. Several students identified Canvas, specifically, as a problem for them.

(5) Active commitment to dedicated time in the course

Student commitment to spending a dedicated number of hours in the course did show some differences across grade levels, with students who had committed to more time spent on the course generally doing better in the course. The chart below shows these variances.

(6) Decrease in perceived and actual workloads

Although the workload was decreased from the pre-redesigned class, students had nothing to compare it to except their own expectations, so the workload still appeared “high” to these students. Some students also did not understand that resubmission of most work was optional, interpreting it as additional required work. For example, one student’s response to the workload was as follows:

“Instead of moving on the next assignments, we were busy correcting previous assignments and it was hard and discouraging to move forward to new assignment.”

However, other students found the workload not only manageable but useful in keeping them focused:

“I feel like this course made it easy as… there was enough work to keep us engaged.”

“This is beneficial in the sense that we were able to see how to improve our assignments and actually get feedback from our work.”

(7) Increased avenues for personal connection with the professor

• Jing: Using Jing to give video feedback on some assignments did not seem to do much for the online students (although I received good feedback on it from in-person class students several times).
• Blackboard emails/announcements: Although I sent out announcements to students through Blackboard once per week or so until after spring break, that did not work very well either. Students did not seem to respond to these messages at all.
• Remind: Having text-message communication as an option through Remind did seem to help, although some students initially interpreted class-wide announcements as personally aimed messages. (I circumvented that in later text messages by always starting them “Hey all,” or “Hi all,” to indicate that they were aimed at the entire class, not individuals.)

SLO Results

The SLOs for this class are listed below.

1. Students will recognize, identify, and explain the theoretical perspectives, main concepts, trends, and historical contexts of the sociological investigation of crime
2. Students will describe the changing roles of policing, corrections and courts in the management of crime
3. Students will analyze the influence of crime on communities and communities on crime
4. Students will analyze and critique the effectiveness of crime prevention policies
5. Students will apply theoretical perspectives to explain current events and crime data
6. Students will draw connections between course content and the world they live in, and propose solutions for social issues relating to crime and policing
7. Students will learn about their own learning process, and apply this knowledge in effective ways to increase their ability to absorb, retain, recall, and apply information
8. Students will be able to identify and differentiate between valid and invalid information sources for use in research and analysis of crime trends and other criminological data

How SLO Success is Evaluated

Evaluating the success of SLOs must take into consideration what constitutes acceptable work in a course.

Under the standards-based grading system used in the course, the target score for homework and module quizzes was 70% or better (if it was below 70%, it did not count towards the grade). Students were also not required to attempt every assignment, but only enough assignments to meet the standard for the letter grade they were aiming for. With this in mind, an average score at or near 70% on these assignments indicates that an SLO was competently completed, and an average score above 70% indicates that the SLO was mastered.

The EMRU system was used to score the research paper projects. EMRU scores were evaluated using a rubric, and based on the percentage range of the total score for the rubric. Rubric scores below 50% earned a U, or “Unsatisfactory.” Rubric scores between 51% and 69% received an R, for “Revision Needed.” A score of at least 70% earned an M, for “Meets Expectations,” while scores of 85% or better earned an E, for “Exceeds Expectations.” (These percentages were not disclosed to the students, in order to make sure that their focus was on quality of work, rather than number of points.) If a majority of the assignments submitted using these requirements received an “M,” the SLO was competently completed. If a majority of the assignments submitted received an “E,” this indicates mastery of the SLO in question.

SLOs 1 and 2
Evaluated by: video-homework, module quizzes, collaborative notes

SLOs 1 and 2 were about basic course knowledge: being able to recognize, define, describe and explain this knowledge summarizes the goals of these two SLOs. I used the collaborative notes assignments, module quizzes, and video homework assignments to evaluate the success of SLOs 1 and 2. Only attempted assignments were included in these averages. The chart below shows the averages for the entire class, and broken down by final class grade range.

As this chart demonstrates, the class as a whole achieved nearly 70% or better on the assignments that measured their basic understanding of course content, even when the entire class’ scores were averaged together.

The collaborative notes assignments are a qualitative gauge of how well the students understood and could explain the core concepts of the class. Some examples of student entries in the collaborative notes assignments follow below.

“Absolute deterrence is keeping people from absolutely never doing anything wrong. Restrictive deterrence is trying to keep people from committing crimes that are labeled severe, such as murder and rape. Specific deterrence is aimed at the specific people committing the crime and focuses on triggering the personal and vicarious experience of punishment in order to persuade people to avoid being punished. General deterrence is aimed at people who would not offend in the first place because of their fear of being punished.”

“Life-course persistent criminality is the looks at what biological traits life long criminals have in common such as head injuries, lead exposure, or oxygen deprivation at birth. Other social traits such as impulsivity, adventure seeking personalities, short temper, or irritability must be taken into consideration as well when looking at life-course persistent criminality.”

“The five reactions to strain theory are conformity, ritualism, retreatism, rebellion, and innovation. Conformists accept the goal and means, even though they cannot get there. Ritualists know that the goal is unreasonable, but keep doing the means because they see no other way of living. Retreatists reject both the goals and the means, so they remove themselves from society. Rebels substitute the goals and means and make their own. Innovators are the criminals, meaning they accept the goal but reject the means to attain the goal.”

“Burgess’ concentric hypothesis described American urban growth to the 1970s.This model was broken down to illustrate 5zones starting from the center to outward.  The center was the central business district and moving outward was the transition zone, workers zone, residential zone and the community zone. This model shows that the closer you live to the central business district, which is the core of the model, the less desirable the homes are and the more likely you are to be accessible to crime.”

Each of these entries is clear, concise, and complete – the standard used to determine if they were acceptable entries to receive pass/fail credit on the assignment. This is just a sampling; there were many entries similar to these in all five collaborative notes assignments.

SLOs 3 through 6, SLO 8
Evaluated by: research paper project set

SLO 7
Evaluated by: research paper project set, writer’s workshops, other workshops

SLOs 3 through 6 and SLO 8 focused on the students’ ability to use their knowledge to analyze criminal behavior and crime policy, apply that knowledge to current events and crime data, demonstrate the use of critical thinking, and show that they could find and use peer-reviewed sources in these processes. I used the research paper project set (annotated bibliography, literature review, and research paper) to evaluate these SLOs.

SLO 7 focused on the student’s ability to learn about their own learning process. The writing workshops and other learning workshops were central to this, and in part, these processes were demonstrated through written assignments. I used the workshops and the research paper project set to evaluate this SLO as well. An additional, separate analysis of SLO 7 will follow the analysis of SLOs 3-8 as a group.

For the annotated bibliography, students were evaluated on four issues:

• how well they created an argument about their chosen criminology-related topic (SLO 3)
• whether the sources they chose to support the argument were valid and peer-reviewed (SLO 8)
• how well they wrote their annotation paragraphs to explain why the sources would help support the argument (SLO 7)
• how well they conformed to ASA formatting on their citations (both inline and work cited forms) (SLO 8)

For the literature review, students were evaluated on five issues:

• how well they created an argument about their chosen criminology-related topic (SLO 3)
• whether they applied the sources they used in the annotated bibliography to the argument by identifying the shared ideas across each source as it related to the argument (SLO 4)
• whether their writing was proofread and well-written or not (SLO 7)
• how well they conformed to ASA formatting on their citations (both inline and work cited forms) (SLO 8)
• how well-organized the paper was, based on the organization of the original argument (SLO 8)

Finally, for the research paper, students were evaluated on eight issues:

• whether the student peer-reviewed someone else’s paper (SLOs 7 and 8)
• how well the student understood and applied their argument to a social problem (SLOs 5 and 6)
• how well the student identified an audience for the argument and structured the paper to appeal to that audience’s needs (SLO 6)
• how well-organized the paper was, based on the organization of the original argument (SLO 8)
• whether the sources they chose to support the argument were valid and peer-reviewed (SLO 8)
• whether the student provided solutions to the problems identified in the paper (SLO 6)
• whether their writing was proofread and well-written or not (SLO 7)
• how well they conformed to ASA formatting on their citations (both inline and work cited forms) (SLO 8)

The chart below shows the total E, M, R, and U scores for attempted paper project set items.

As this chart demonstrates, nearly all students who submitted any of the paper project set items scored in the “meets expectations” or “exceeds expectations” category; only 5 of the 35 attempts did not meet these standards. This indicates that SLOs 3 through 8 were either met or mastered.

Although the collaborative notes were originally expected to work as an evaluator for  SLOs 3, 4 and 6, they did not often move into analysis, application, or critical thinking, so they are not included as part of this evaluation.

SLO 7
Evaluated by: writer’s workshops, student success workshop, other workshops

SLO 7 focused on meta-learning beyond the course content: helping the student learn how their learning process worked, as well as giving procedures to help with big projects such as the paper project set. As these workshops were scored complete/incomplete, there is no quantitative evaluation available. In order to allow a qualitative overview of the effectiveness of these workshops, student comments from these exercises appear below.

Successful Student Workshop

This workshop is composed of several topics: time management, study skills, and classroom participation. The discussions took the form of answers to structured worksheet questions. Example questions and student answers to those questions are provided here.

Question (study skills and classroom participation worksheets): List five to eight steps you will take in working toward your goal (set in previous questions).

Student responses to this question included:

1. Make sure to minimize the noise in the area I am in if I can.
2. Don’t go on social media while I am studying
3. Make sure my phone is on silent, or Do not disturb if absolutely necessary
4. Set goal times without stopping what I am doing. Ex. Timer for 20 minutes without breaks.
5. Change locations of studying

1. I will set aside at least 2 hours to study for my exam day for a week before test.
2. Organize all my notes and study material that I will need.
3. Make flashcards on the important information.
4. Each day study the flashcards.
5. Make note of the flashcards I need to study more.

1. I will put reminders on my phone of all due dates.
2. I will put some reminders on my calendar as well.
3. I will have paper next to me when I am doing readings.
4. I will find a way to make it easier for me to take notes when reading.
5. I look for ways to take notes easy but effective.

1. Wake up early and on time; do not oversleep.
2. Make sure to have breakfast to have energy for the day.
3. Leave 10 minutes earlier; in case of an emergency or traffic.
4. Drive safely to avoid getting pulled over or an accident.
5. Go straight to class and do not stop and chat with friends.

Although many of these steps may seem obvious, some of the students found this process to be an eye-opener. Students often mentioned that it had never occurred to them that being hungry or being short on sleep might be part of why they struggled in school. Others had never learned basic study habits such as taking notes while reading, or rewriting their own notes in different formats, until they took the workshop.

Question (time management worksheet): Name three strategies you will apply from this workshop to your own approach to time management, and explain how you will put them into action.

1. Backwards-forwards method is something I want to try out with task that seem to overwhelming. This will help me see the progress I’ve made and will encourage me to keep going through the steps.
2. I think I could benefit from using a wall calendar, I usually rely on my planner but I think having a bigger picture of things I need to complete can be helpful.
3. At times, prioritizing things can become difficult. This is not a very big issue that I struggle with but there are times that I would choose doing something fun when I could be getting tasks done. The Eisenhower box is something that I’ve used before so I can utilize it more.

1. When I am tired I will make sure I get enough rest before I begin my assignments. Getting rest will improve the accuracy of each item I am working on.
2. I will learn to deal with procrastination by taking the advice in this workshop. I will use willpower to make sure I am forcing myself to sit and complete my work.
3. I will not multitask. I will work on one assignment at a time.

1. I will use the to do list as a strategy to manage my time not only with school, but my work and personal life.
2. I will break down my goals into smaller steps so that it doesn’t become overwhelming or crammed.
3. I will make reminders by writing them down instead of just reminding myself.

These responses show that many students struggle with procrastination, forgetting assignments, and other issues that interfere with good time management. The workshop serves as a starting point to help them overcome these issues, by at least making them visible to the students.

Writer’s Workshop 1

Writer’s Workshop 1 is an overview of writing and the myths, excuses, and fears students have about it. In several places, students have the opportunity to reflect on the workshop as it relates to their own lives and experiences:

“My biggest stressor with writing is when I have a deadline. I believe creative writing takes time but having a deadline makes it difficult to focus. Another stressor for me when it comes to writing is not having a guideline or rubric. Knowing who my audience is and what my reader is looking for makes writing easier for me.”

“One thing I feel I will take into consideration when writing a paper is to brainstorm what I want to discuss and come up with ideas. Another thing I want to do is get used to working up drafts and not going directly to making my final paper. This will allow me to improve my paper and see my progress instead of just going directly to writing. “

“I’ve always been creative to a degree but always had issues implementing or producing an adequate paper. Hopefully, with my new found discoveries on the writing process, how to conduct research, study and take notes I will improve which will boost my confidence. What stood out most and was very surprising is that writer’s block being described as being a form of avoidance.”

“I learned from this workshop to take it step by step. I will actually use this workshop when writing my papers to help guide my through my paper. It helps to not get lost when writing because sometimes your mind wonders (sic) when you think too hard or too much.”

Writer’s Workshop 2

This workshop included an exercise in avoiding plagiarism. Some of the student reflections on this exercise appear below.

“This exercise taught me how important it is to be aware of plagiarism when writing. Even the slightest misuse of someone else’s work is considered plagiarism. This is a serious academic issue that has serious consequences, including being expelled. I learned that citing is a very crucial part of writing in order to avoid plagiarism.

“The one way to avoid plagiarism is to always make sure to credit other people’s work when using it. This means including in-text citations along with a citation in your works cited page. Not only that, but do not copy a person’s work word-for-word unless it’s a direct quote. Even then, it should still be cited. This is something every student needs to incorporate in their steps when writing a paper.”

“It is important that we cite any and every piece of work that is not our own. One thing that I learned is that sometimes we can do it unknowingly, but that does not mean that we care exempt from the consequences of it. One way to avoid plagiarism is to cite as you go so you do not forget to cite later.”

Writer’s Workshop 3

Writer’s Workshop 3 goes into some detail about how to create and proofread a second draft from a rough first draft. The issues covered include spelling, grammar, organization, citation, overwriting, and other typical student writing issues. Some student reflections on this workshop are shown below.

“I have some trouble keeping my thoughts together which causes me to go off topic often and then lack focus. As far as the $12 word syndrome problem, I started originally as an English major and my mother was an English major as well. I have always been used to using big words and fancy language or “fluff” to describe things as vividly as possible. I could fix this by going through my drafts and picking out any unnecessary words.”

“I think I have these types of problems because I didn’t know how to properly cite. The word count, and minimum page requirement gives me a lot of anxiety and I just start writing to fulfill the requirement not for substance. I need to take my time and go through the proper process of writing a research paper. This means writing down all the information on the topic, writing out my reference page, and slowly constructing the paper from there.”

“I think these problems are mostly made just because I don’t really know the correct way. This workshop helped me see that these issues are what gets my papers marked down and the workshop gave me the solutions or corrections that need to be made. Now that I am aware of the corrections I can make them. The way I can make these corrections is just being aware of them first so I can fix it.”

Research Handout

This workshop talks about the basics of finding and using research, what an argument is (and what it isn’t), and how to use the library resources. Similarly to the Successful Student Handout, this was structured with worksheets that had questions. Popular answers to the question “Before you took this workshop, which of the following misconceptions did you have about research?” included:

Focusing on page or word counts
Not visiting the librarian for help
Using minimal sources
Writing the paper before looking at the research
Using non-scholarly sources

Steps students identified that they would take to deal with these issues in future included:

• I will remember to use the librarian as a source to help better my sources.
• I will use more than the amount of minimal sources in order to benefit my paper.
• I will use scholarly sources when doing a paper to get the best paper I can out of it and the correct facts.
• I will make time each day I work on my paper to go to the librarian.
• I will focus on maintaining a strong argument to distract me from the page or word count.
• I will keep my research organized while taking notes when reading through them.
• I’ll start my search in the library
• I’ll make sure I carry sticky notes and paper to write notes on
• I’ll use more physical sources like books instead of just websites

Success Compared to Prior and Concurrent Classes

The chart below compares the pre-designed section, the in-person section, and the redesigned section across all grade outcomes.

In the pre-designed course, there were no A scores at all, and the DFWI rate was 60%.

In the in-person redesigned course, 5 students earned As, and the DFWI rate was 17%.

In the online redesigned course, 4 students earned As, and the DFWI rate decreased by 2% to 58%.

The in-person class far outpaced the fully-online classes in terms of success. The stark difference between the online and in-person courses for Spring 2018, which have the same essential course design, assignments, and requirements, indicates that part of the problem may simply be that the redesigned course is a completely asynchronous online course. We may need to re-evaluate whether that is a valid choice for this course. There are other reasons to consider making the class at least partially synchronous, which will be discussed in the final reflections and recommendations.

Unexpected Results

The most unexpected result is the large difference in outcomes between the in-person class and the fully-online class. A scheduled and regular class meeting time, the presence of the professor, and co-presence with classmates apparently makes a far greater difference to student outcomes than would be expected. Some online student responses to the question “What would have helped you learn more effectively in this class?” included:

“More discussions with piers (sic)”
“If it was in class, face to face”
“More interaction with the professor”
“Lecture in classroom”

Although many of these students took an online class, they seem to have expected it to be more like an in-person classroom than was possible with a fully-online, completely asynchronous setup.

Student Feedback

13 students responded to a student survey sent out at the end of the semester. The survey asked about the EMRU scoring system, the standards-based grading system, which assignments helped most with learning, which assignments did not, feedback on what would have helped the student learn more effectively and what did not help them learn, and a request for suggestions of up to three ways to help students in online courses stay engaged and participate regularly.

Grading Systems

Five of the students reported a positive experience with the EMRU scoring system, and six reported a neutral experience with it. Only two of the 13 respondents reacted negatively to the scoring system.

Student comments:

(Positive) “This system was easy to understand and straight to the point of what was expected to reach each score.”

(Neutral) “I liked knowing exactly where my grade stands in the class.”

(Negative) “[I]t was hard to actually know how well you did or not.”

Five of the students reported a positive experience with the standards-based grading system, and five reported a neutral experience with it. Only three of the 13 respondents reacted negatively to the grading system.

Student comments:

(Positive) “It helps relieve the stress that tends to come with school and I’ve found it to work very well for me.”

(Positive) “I think the fact that wasn’t stressed about assignments that I knew I didn’t need to do because they wouldn’t effect (sic) the grade I want helped me learn effectively because it made me much more focused on the tasks I needed to do.”

(Neutral) “This is beneficial in the sense that we were able to see how to improve our assignments and actually get feedback from our work.”

(Negative) “Everything we submitted, it was sent back for corrections and it was frustrating to keep getting our work kicked back.”

Assignment Types That Helped

The top-voted assignment types that students found helpful were the lecture-video/homework assignments and the module quizzes. There was a mixed reaction to the workshops, goal-setting assignments, research assignments, and collaborative notes – some students found them helpful and others did not. The students did not find the threshold quizzes or the syllabus discussion board (both gatekeeper assignment types) to help with their learning.

Lessons I Learned/Reflections

Planning and Designing Issues

If you have set up an assignment to be a “threshold” on other assignments, such that the later assignments cannot be accessed until the first assignment is done, it’s best to avoid a due date on the “threshold” assignment. If you put a due date on it, and the due date passes, students are essentially locked out of the entire course. Feedback from the students also indicated that they do not find the threshold quizzes helpful but frustrating, although part of this may have been issues that appeared in Canvas that interfered with them working correctly.

Making sure that the LMS is on the same time zone as the school is essential. I wasn’t made aware of the fact that the Canvas LMS defaults to Mountain time until the middle of week 3. As a result, I had to edit and grade quite a few assignments that students thought in good faith they had turned on time. Students were still reporting issues with this well into week 15.

The pre-design elements may have helped reduce the post-census DFW rate, but not as much as I would have liked. It would help if the CSU system would give instructors three or even four weeks, rather than only two, to initiate instructor drops. Two weeks really is not enough time to assess whether students will be able to complete the course.

Technology

Using new technological tools always requires a learning curve – both for the students and for the instructor. Although Canvas did address what I felt was a big issue with Blackboard – not showing the students what they had to do to get to the next activity – I still had a steep learning curve. Some of the things I expected Canvas to do did not happen the way I had assumed they would, and in the middle of the semester, conditional release simply stopped working. This added quite a few time demands and unpleasant stress to my life. In the future, I will find a way to use Blackboard to do conditional release that tells the students what they need to do to move to the next step (perhaps a comment below the assignment link), if I decide to use it at all.

Sometimes technology also creates more work than it saves. Having the course spread out over Canvas, PlayPosit and Remind confused some students, and it created more work for me. I could just as easily have embedded the lecture videos into Blackboard with attached homework questions, had the students answer them in Blackboard, and graded them in Blackboard with a rubric that would have allowed me to comment on their answers. This will be my procedure for flipped and fully-online classes from now on.

Of the technology tools I used in this redesign, I will continue using Remind, Zoom, and Jing. Remind allowed me to “reach out and touch” the students in a way that I knew they would see – their cell phones. Zoom, although my students didn’t use it often, was occasionally critical in making sure I could discuss an issue with a student instead of having a long back-and-forth chain in email. And Jing allowed me to give audio and visual feedback to the students, which some of them found very helpful in their revisions.

However, I will no longer be putting my lecture videos on PlayPosit, because their system has a number of issues that make grading less efficient and less effective. An instructor cannot download a single class’ information; it downloads every class in one large .csv file, requiring extensive editing and rearranging. Playposit also does not notify the instructor when there are new submissions, and some students find the interface difficult to work with.

Similarly, I will not be using Canvas again, because in addition to the conditional release problem, it has a number of issues with grading, test creation, and notifications that Blackboard does not have. I will be returning to Blackboard after this semester.

Grading System and Due Dates

The grading system turned out to be an issue, so I am working towards simplifying it even more than it is already. I still want to bring in the iterative process of learning and focus more on the research assignments (annotated bibliography, literature review, and research paper), so I will be re-thinking the grading system over the summer for my fall classes. This includes re-thinking whether due dates are even useful, beyond “you must complete at least two items per week” as attendance, to keep the students focused on the class and their work.

Although I find it superior to a percentage-based grading system and research supports its use, I will probably no longer use a standards-based grading system with fully-online courses, unless I am able to teach those courses with synchronous class meetings. Many students find it difficult to understand written instructions, and need the live/in-person auditory back-and-forth that only an in-person or synchronous class can provide.

Adjustments

In the last two weeks of the semester, it became clear that the grading system needed to be adjusted due to the large numbers of students who were not completing work. Instead of requiring the students to complete all the assignments labeled “Basic” in the grading system, I assessed what had been done, and adjusted the “Basic” requirements to 13 video-homework assignments at 70% or better, three module quizzes at 70%, and one completed Collaborative Notes assignment across the board for all grade columns, making the research projects the conditions for the A, B and C course grades. On May 1, I held an online video meeting in Zoom for the students, to go over this adjustment and the annotated bibliography requirements, in the hopes that more students would submit this make-or-break assignment (many still had not). I also provided a recording of the meeting to all students who could not attend.

Student Interaction and Engagement

Allowing students to have a voice in decisions is critical. At the end of week 4, I sent out a poll to both of my online classes, proposing that the due dates for the assignments in each module all be changed to the last day of the module, giving students more freedom to submit their assignments (while still requiring two assignment submissions per week for attendance). The students overwhelmingly voted to make this change.

Attendance Issues

The charts above demonstrate one of the big problems with fully-online classes: reckoning attendance. The correlation of absences to grades is obvious: students who attend regularly get better grades. Because this is an online class, attendance was measured by attempting at least two assignments per week (the student did not need to get a passing grade for an assignment to count towards attendance). Either we need a different way to measure attendance, or we need to find another way to motivate students to attend. The current method does not seem to work for many students.

One of the common assumptions about online classes is that they should be designed to be 100% asynchronous whenever possible, as students depend on them being more flexible than in-person classes. However, it appears that most undergraduate students who are not prepared for online classes need regular structure in the form of some kind of class meeting. Based on these outcomes, and the comparison to the nearly-identical in-person class (which had drastically different results), I recommend that online classes have synchronous online class meetings, so students have regular interaction with the instructor and each other, and so they are reminded that they are in a class due to required class meetings.

The synchronous meeting does not have to be a long meeting. It could be a required office hour, where the instructor opens a Zoom room for a set period of time each week and requires students to log in for five or ten minutes during that period of time. This would allow for student-to-professor and student-to-student interaction in real time, and fix the problem of reckoning attendance.

Findings and Recommendations

• Preparation: Recognizing that many students are not prepared for the time management, time commitment, or asynchronous format of online courses is critical. A colleague at USC told me that undergraduate students at USC are not allowed to take online courses due to these issues.
  • Students should receive an in-person course about how to succeed in an online course before taking one, and this course should be a prerequisite for taking online courses. It should cover time management, time commitment, reading instructions carefully, using the technological platforms, and other issues.
• Advising Issues: Faculty need access to students’ academic advisors, and these advisors should be notified any time a student stops making satisfactory progress in an online course.
  • A mid-term report requested by advisors on online students, similar to the mid-term reports requested by the athletics department on student athletes, would be a good idea. Online instructors often have no way to reach students except for easily-ignored methods like email, while advisors may have more avenues.
• Instructor Drop Ability: Faculty must be given the ability to drop students who are not making satisfactory progress beyond the two-week instructor drop period. The number of WUs I have due to students just not showing up/logging in anymore is excessive.
• Synchronous Meetings: Online courses must have at least one synchronous meeting required per week. There are two reasons for this:
  • Student Accountability: Students are used to having a set class meeting time that reminds them that they have a class to prepare for. Without this reminder, many students simply stop attending the class.
  • Attendance and Workload: Accreditation and financial aid policies both require we demonstrate students are attending class. Right now, the only way to measure attendance in a fully-online, asynchronous course is to require the student to turn in at least one assignment a week. This is a minimum of 15 assignments over the semester, which many students find an onerous and impossible workload. A once-per-week required synchronous meeting would allow us to demonstrate attendance for these purposes without basing attendance on items turned in. It would also remind students to attend class.

Reflection on Participation in the CRT Institute

The week I spent at the CRT Institute in June of 2017 was quite possibly the most immersive academic experience I’d been in since graduate school. Making connections with peers outside of my specific discipline sparked new ideas for me. I still stay in touch with many of the people I met during that week. Knowing that I wasn’t the only one confused by some of the technology we were trying to learn was also a relief.

Working with a cohort that was working to create fully-online courses has helped me. Reading over Danielle Hidalgo’s information about how to keep students engaged, for example, gives me ideas for how I will work to keep students engaged (I just have to get over my fear of videos that show my face, that’s all). Talking with Allison Evans, my cohort leader, helped me a lot when I didn’t feel like anything I did was working.

The regular Zoom meetings over the past academic year, both with my cohort and the wider PLC meetings, have helped me with a wide range of issues as well, including how to manage Merlot and how to make sure my course videos were accessible. One cohort meeting in particular has stuck with me since it happened: our leader brought in two instructors who had designed a required fully-online course, and the advice they had for recording course videos blew my mind. I spent the next two weeks working on revising and re-recording many of my course videos to make them shorter and clearer, and my students this semester told me that their parents thought they were listening to podcasts when they overheard the videos (a huge compliment!).

Building this ePortfolio has also been an enriching experience for me. The guidelines provided by the CRT team helped me understand what information would be most valued and useful.

In the future I will only introduce one change at a time and see what works, instead of throwing all the changes into the mix at once. I feel that was my biggest error in this redesign. This redesign has also taught me that I tend to get over-excited and try everything I’ve read or heard about, when I should be focusing on one or two effective methods or changes and how to maximize their outcomes.

My only regret is that I am in the last cohort of CSU faculty to participate in this program. I wish it could be continued so that I could have these experiences again and bring the learning back to my students.

– fin –