Building SkillBlox as Design and Development Research

We are building new features for SkillBlox as part of our partnership with the American Institutes for Research (AIR) in IES-funded design and development research. Our qualitative approach is guided by Design Based Research (DBR) (Brown, 1992; McKenney & Reeves, 2018; Sandoval, 2014), through which we’re able to learn from end-users what they want to see in the new features, especially in the user experience and user interface (UX/UI). Just as important, DBR gives us a process to elucidate findings about teaching and learning more broadly by establishing, revisiting, and updating conjectures. A conjecture is the research team’s best guess as to how elements of design impact teaching and learning (Sandoval, 2014).  Ultimately, the goal of the research is to deliver useful and relevant SkillBlox features, continuously refine our conjectures, and, along the way, answer our research questions about how SkillBlox can support expanded TSTM-aligned instruction. 

• What SkillBlox features best support the use of Open Education Resources (OER) in TSTM-aligned instruction? 

• What level of description and tagging of resources do teachers identify as helpful? 

• What mechanisms for prioritization and representation of OER streamline the OER search and selection process? 

• What user interface and user experience best promote use of SkillBlox?

This work is happening in three distinct phases, which are in various stages of completion at the time of writing: 1) exploratory work, 2) iterative feature development and testing, and 3) a usability and feasibility study. 

Phase 1: Exploratory Work.

Our exploratory research, concluded in late 2022, included a review of prior research on the use of OER in various education settings, previous SkillBlox development and accompanying data collection, and TSTM Implementation research led by AIR. Through these sources, we gleaned some guiding information:

Prior research on the use of OER in adult basic education is thin (Newman et al., 2015), but the literature does articulate both barriers to OER use and strategies that support it. 

Past development of SkillBlox, funded by the World Education Services Mariam Assefa Fund (no connection to World Education), showed us that users: need to have hubs to support SkillBlox sharing, want to know the level of the activity and the context for how it is used, and prioritize being able to select the language (if the resource that has multiple language options).

In the TSTM implementation research led by AIR, teachers noted that they need access to granular instructional resources and activities aligned to the different TSTM skills that they could then adapt for their own use. However, they also reported having limited time and capacity for this kind of work. To partially address this problem, they welcomed opportunities to collaborate with others. 

Collectively, these observations informed the development of our initial DBR conjecture:

Phase 2:  Prototype and Iterative Feature Development and Testing.

This phase is happening in two stages: prototype testing and iterative building and testing. Our prototype testing began in early 2023. Over several months, we gained insight about our initial designs through focus group meetings, questionnaires, and participant engagement in SkillBlox. Our focus group discussions, held as three meetings with six volunteer educators, were especially fruitful. We employed a qualitative coding process to examine notes and transcripts from those meetings, and triangulated our observations with what we learned in our exploratory work, especially the implementation study introduced above. We have also gleaned information on prototype design from our EdTech Maker Space data analysis. Collectively, these data and analyses supported our original conjecture and resulted in a prioritized features list that responds to both the research questions and our design goals. 

At the end of this cycle of iterative design and testing, we will have a complete set of newly developed features and training materials needed to prepare teachers for using SkillBlox in TSTM-aligned instruction. 

Phase 3:  Usability and Feasibility (U/F) Study.

Starting in June 2024 we will pilot these new features and the training with 10 teachers and their learners. The U/F study will be the first opportunity to observe real use of the application’s new feature sets and test our onboarding process and resources. Again, because we are working with DBR, we will be able to not only make improvements to both the application and resources, but we will gain insight into teachers’ use of OER and SkillBlox to support teaching and learning. 

Future Research

The IES-funded study will conclude with an experimental pilot study of the new SkillBlox features and PD activities. The research, conducted in 2 – 4 states beginning in fall of 2024, will investigate whether the PD and other supports are sufficient for teachers using TSTM-SkillBlox, and whether teachers find the new features and content of SkillBlox easy to use and adapt in their instruction. The pilot will also identify remaining challenges teachers experience in implementing TSTM-aligned instruction using the application and elucidate needs teachers have for additional support to be able to use TSTM-SkillBlox easily.

References

Admiraal, W. (2022). A typology of educators using open educational resources for teaching. International Journal on Studies in Education, 4(1), Article 1. https://doi.org/10.46328/ijonse.60 

Belikov, O. M., & Bodily, R. (2016). Incentives and barriers to OER adoption: A qualitative analysis of faculty perceptions. Open Praxis, 8(3), Article 3. https://doi.org/10.5944/openpraxis.8.3.308 

Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141–178.

Hilton III, J., Wiley, D., Stein, J., & Johnson, A. (2010). The four ‘R’s of openness and ALMS analysis: Frameworks for open educational resources. Open Learning: The Journal of Open, Distance and e-Learning, 25(1), 37–44. https://doi.org/10.1080/02680510903482132 

Marín, V. I., & Villar-Onrubia, D. (2022). Online infrastructures for open educational resources. In Handbook of Open, Distance and Digital Education (pp. 1–20). Springer Singapore. https://doi.org/10.1007/978-981-19-0351-9_18-1 

McKenney, S., & Reeves, T. C. (2018). Conducting educational design research. Routledge.

Newman, A., Rosbash, T., & Sarkisian, L. (2015). Learning for life: The opportunity for technology to transform adult education. Boston, MA: Tyton Partners. http://tytonpartners.com/tyton-wp/wp-content/uploads/2015/03/Learning-for-Life_The-Oppty-for-Tech-to-Transform-Adult-Education_March-20151.pdf 

OER: Adult Learning and Sustainability. (2022, March 21). EPALE – European Commission. https://epale.ec.europa.eu/en/blog/oer-adult-learning-and-sustainability

Promising practices and lessons learned from early implementation of the TSTM framework (2023). American Institutes for Research. Create Adult Skills Network. https://createadultskills.org/index.php/resource/promising-practices-and-lessons-learned-early-implementation-tstm-framework 

Sandoval, W. (2014). Conjecture mapping: An approach to systematic educational design research. Journal of the Learning Sciences, 23(1), 18–36. https://doi.org/10.1080/10508406.2013.778204 

Taylor, C., & Taylor, M. W. (2018). I’m never doing this again!: Identifying and solving faculty challenges in adoption of open educational resources. Online Journal of Distance Learning Administration, 21(2).

United Nations Educational, Scientific and Cultural Organization (UNESCO). (2019). Recommendation on Open Educational Resources (OER) | UNESCO. https://www.unesco.org/en/legal-affairs/recommendation-open-educational-resources-oer 

Wolfenden, F., & Adinolfi, L. (2019). An exploration of agency in the localization of open educational resources for teacher development. Learning, Media and Technology, 44(3), 327–344. https://doi.org/10.1080/17439884.2019.1628046