Learning Management System (LMS) Evaluation - Week #7
LMS Evaluation Introduction
Executive Council Welcome
Council Members – Welcome to the final recommendation for the implementation of a new Learning Management System (LMS). This executive summary provides a brief comparative analysis used for evaluation of two LMS software applications: Brightspace (formally known as Desire to Learn or D2L), and Canvas by Instructure. Also presented is the current LMS state, a summary of LMS analysis used for final determination, and final recommendation for implementation.
Executive Council Welcome
Council Members – Welcome to the final recommendation for the implementation of a new Learning Management System (LMS). This executive summary provides a brief comparative analysis used for evaluation of two LMS software applications: Brightspace (formally known as Desire to Learn or D2L), and Canvas by Instructure. Also presented is the current LMS state, a summary of LMS analysis used for final determination, and final recommendation for implementation.
Problem Statement - Background
On June 27, 2017, I was directed by the President to perform detailed comparative analysis of two industry leading Learning Management Systems. This direction was based on significant findings from student reviews, post-class surveys, analytics derived from the current LMS, and university administrator feedback.
Student satisfaction ratings in post-class surveys across all university colleges indicated ratings of “dissatisfied” or “very dissatisfied” using Likert-type survey instruments. Survey narrative feedback revealed the top three student complaints included a lack of user-friendly design on the welcome page, and poor or unfriendly accessibility features for students with disabilities.
The President also noted administrator dissatisfaction. Professors frequently complain about system use, customer support, and frequent downtime. The University Chief Information Officer recently wrote a white paper highlighting metrics that related to current system functionality, and noted that current LMS costs per student are becoming unsustainable.
On June 27, 2017, I was directed by the President to perform detailed comparative analysis of two industry leading Learning Management Systems. This direction was based on significant findings from student reviews, post-class surveys, analytics derived from the current LMS, and university administrator feedback.
Student satisfaction ratings in post-class surveys across all university colleges indicated ratings of “dissatisfied” or “very dissatisfied” using Likert-type survey instruments. Survey narrative feedback revealed the top three student complaints included a lack of user-friendly design on the welcome page, and poor or unfriendly accessibility features for students with disabilities.
The President also noted administrator dissatisfaction. Professors frequently complain about system use, customer support, and frequent downtime. The University Chief Information Officer recently wrote a white paper highlighting metrics that related to current system functionality, and noted that current LMS costs per student are becoming unsustainable.
LMS Executive Summary and Final Recommendation
Evaluation Process of Learning Management Systems
The hands-on, first person evaluation of Brightspace and Canvas tested 25 functional attributes of each system (50 total evaluations) over a five-week period. Attributes evaluated were identical or near-identical, with the goal of achieving the same result for each system. Week 6 evaluation reviewed five “soft” areas, known as a behind the scenes evaluation based on first-person feedback from IT administrators who evaluated both Brightspace and Canvas for their implementation into their own organizations. Cursory internet research was also used, but did not lend significant credibility to the behind the scene analysis.
Evaluation Process of Learning Management Systems
The hands-on, first person evaluation of Brightspace and Canvas tested 25 functional attributes of each system (50 total evaluations) over a five-week period. Attributes evaluated were identical or near-identical, with the goal of achieving the same result for each system. Week 6 evaluation reviewed five “soft” areas, known as a behind the scenes evaluation based on first-person feedback from IT administrators who evaluated both Brightspace and Canvas for their implementation into their own organizations. Cursory internet research was also used, but did not lend significant credibility to the behind the scene analysis.
Findings
Each week’s analysis concluded in an overall weekly recommendation. In conclusion, Canvas yielded a 66.7% final recommendation over Brightspace. While this percentage doesn’t lend itself to a landslide win for Canvas, several positive recurring themes emerged from the analysis.
For Canvas, these themes included initiative use requiring little or no need to reference tutorials, significantly less navigation to accomplish similar tasks than Brightspace, course customization options, ease of grading (referred to as “Speedgrader”), outstanding 24-hour customer service, a guaranteed system up-time of over 99.95%, availability of Canvas across mobile devices, and lower cost per student.
Supporting behind-the-scenes evidence included one IT administrator’s contention that overwhelming positive feedback from their administration and students drove their decision it implement Canvas, not cost.
Final Recommendation: Implement Canvas by Instructure to replace the existing Learning Management System.
Each week’s analysis concluded in an overall weekly recommendation. In conclusion, Canvas yielded a 66.7% final recommendation over Brightspace. While this percentage doesn’t lend itself to a landslide win for Canvas, several positive recurring themes emerged from the analysis.
For Canvas, these themes included initiative use requiring little or no need to reference tutorials, significantly less navigation to accomplish similar tasks than Brightspace, course customization options, ease of grading (referred to as “Speedgrader”), outstanding 24-hour customer service, a guaranteed system up-time of over 99.95%, availability of Canvas across mobile devices, and lower cost per student.
Supporting behind-the-scenes evidence included one IT administrator’s contention that overwhelming positive feedback from their administration and students drove their decision it implement Canvas, not cost.
Final Recommendation: Implement Canvas by Instructure to replace the existing Learning Management System.
Evaluator Background
Bret Retherford is a 33-year federal employee and retired Civil Engineer Chief Master Sergeant. His current role is Technical Lead and Deputy Division Chief of Facility Management at the National Air and Space Intelligence Center. In this capacity, he oversees strategic planning analysis, implementation, and oversight of Center facility operations while serving in a formal leadership capacity.
Bret is currently pursuing an Educational Technology/Instructional Design Master of Education (M.Ed.) degree at Wright State University (WSU), and is a 2017 WSU Master of Science in Leadership Development (MSLD) graduate. He is also a 2004 graduate of Urbana University with a Bachelor of Science in Business and Organizational Leadership.
Bret Retherford is a 33-year federal employee and retired Civil Engineer Chief Master Sergeant. His current role is Technical Lead and Deputy Division Chief of Facility Management at the National Air and Space Intelligence Center. In this capacity, he oversees strategic planning analysis, implementation, and oversight of Center facility operations while serving in a formal leadership capacity.
Bret is currently pursuing an Educational Technology/Instructional Design Master of Education (M.Ed.) degree at Wright State University (WSU), and is a 2017 WSU Master of Science in Leadership Development (MSLD) graduate. He is also a 2004 graduate of Urbana University with a Bachelor of Science in Business and Organizational Leadership.
Bret in his Instructional Design "Command Center"
Additionally, he holds two Associate of Science degrees from the Community College of the Air Force (CCAF) in Electrical/Mechanical Technology and Operations Maintenance Management.
Professional certifications include: CCAF Professional Manager Certification (PMC), Certified Energy Manager (CEM) from the Association of Energy Engineers, Real Property Administrator (RPA) and Facility Management Administrator (FMA) from the Building Owners and Managers Institute.
Additionally, he holds two Associate of Science degrees from the Community College of the Air Force (CCAF) in Electrical/Mechanical Technology and Operations Maintenance Management.
Professional certifications include: CCAF Professional Manager Certification (PMC), Certified Energy Manager (CEM) from the Association of Energy Engineers, Real Property Administrator (RPA) and Facility Management Administrator (FMA) from the Building Owners and Managers Institute.