Exploring the EdTech Maker Space: Developing and Using our Codebook

In a previous blog post, we introduced you to the research aims and methods associated with the Teaching Skills That Matter-SkillBlox (TSTM-SB) Instructional Support Pilot project. We started introducing the data collection and coding process the World Education research team used to analyze data collected during our EdTech Maker Space (ETMS) professional development events. As a reminder –  that data set included transcripts and notes from our ETMS sessions, transcripts of focus groups with participants and facilitators, questionnaires about Makers’ preferences and skills, and the media that the Makers created. The analytical process was iterative and systematic and designed to help us understand how teachers participated in the ETMS to better make use of and create open educational resources (OER). Now, we’ll take you deeper into our research journey by describing how we have been developing our codebook and making use of it to guide our qualitative data analysis.

Our Codebook

A codebook is a list of defined codes that helps researchers identify patterns and themes reflected in data in a consistent manner. Our codebook includes the following categories of information shown in Figure 1: 

• Name of the code

• Number of the code, which helped us remember the organization of the codes

• Description of the codes, which outlines the qualities or properties of the code (Saldana, 2014, p. 28) 

• Inclusion criteria, the conditions that determine whether an excerpt meets the criteria required for the code

• Exclusion criteria, the conditions that determine whether an excerpt does NOT meet the criteria required for the code 

• Examples of the proper use of the code (not shown)

• “Close but no” example, an example of an excerpt that has information similar to what could carry the code, but does not fully meet the inclusion criteria (not shown)

Defining the Codes

We crafted the code definitions through a collaborative and iterative process to ensure consistency of analysis across the research team members and the accuracy of the codes applied to excerpts of text. In our initial attempt to define each code, we drafted definitions based on the information contained in the data excerpts, keeping in mind our research questions and the project context. The definitions in the code book are useful tools for helping us make decisions about what code to apply to any given excerpt while coding. Although the definitions provide important guidance, they are not set in stone; we have made ongoing refinements by testing each definition with a discussion of how the different research team members used the code. Essentially, as we read more data, we have more information to help us refine our definitions.

For example, we have added inclusion criteria, exclusion criteria, examples, and “close by no” information as needed. This additional information helps us to differentiate between possible codes and affirm our decisions about the content of the data excerpts to be labeled by a code. By discussing code usage and definitions with the team, we have reached a shared understanding and can more consistently apply the codes. We have developed the code book so that our coding has become predominantly deductive as we use existing codes to analyze new data. 

Analysis: Organizing Codes by Themes

In addition to defining the codes, we have organized the codes into groups based on a theme. This has helped us make high-level groupings of data, which we can use to quickly identify text excerpts for analysis. We started with three top-level codes.

1. Experience in the Maker Space. This group contains codes that refer to how we are facilitating the ETMS. The excerpts included as coded data in this group give clues about the inputs that shape participant learning. 

2. Products of the Makers. This group of codes refers to what the Makers are doing, including the tasks of alignment, evaluation, adaptation, and curation. These data refer to the service part of the service learning.

3. About the Makers/Participants. This group of codes is applied to any information about who our research participants are. These data inform us about the Makers’ skills, knowledge, preferences, and motivations. 

We also added second- and third-level codes for each category.  While the top-level code represents a broad category, subsequent levels provide increasing detail and specificity. Here’s an excerpt from our codebook that provides an example of all three levels and their definitions:  

1. Experience in the Maker Space (The how: How we are facilitating the Makerspace). The learning part of service learning

   1. Supports to Makers (Provided to Makers – e.g., via facilitators) or other educators creating, evaluating, etc OER). 

      1. Facilitation – how the participants perceived/described facilitation

Refining the Codes 

Refining the codes involves a continuous ‘conversation’ between the researchers and the data. During the coding process, researchers occasionally add new codes when data does not fit into the existing coding structure. Coding also often prompts questions and discussion among the research team about distinguishing between codes, such as whether a given excerpt relates better to “Products of the Makers” or “Experience in the Maker Space” – or whether similar codes should be consolidated. This collaborative refinement improves our analysis by boosting the consistency in our coding.

The final high-level codes include, 

1. Experience in the Maker Space: describes how the Maker Space was facilitated 

2. Products of the Makers: describes the alignments, evaluations, adaptations, created, and curated products 

3. About the Makers: describes demographic information; motivations; attitudes about various ETMS-related topics; goals; and prior experiences

4. TSTM: identifies information related to the Teaching Skills that Matter framework, and TSTM as it relates to ETMS-driven content, use, and applications

5. OER: OER info as it relates to ETMS driven content, use, and applications. This does not include attitudes about OER which are categorized under “About the Makers”

6. Skillblox Development: identifies feedback about various features within the Skillblox application

Finding Meaning through Coding and Notetaking

Coding helps us to find meaningful themes, trends, patterns, and insights within the data collected. Coding helps researchers understand what is occurring and how participants respond to those occurrences. 

In addition, the notes, annotations, and memos made by the World Education researchers and added to NVivo during the coding process help the team make meaning about what is being coded. These observations made while coding help to raise questions or highlight relevant points of interest that were noticed by members of the research team. This in-the-moment documentation of observations serves as an active dialogue between researchers and the data and helps our team interpret our findings

“Maker Design Decisions/Capacity – and CONFIDENCE:

• Observation from facilitators that comfort level with open-ended nature aligned with the skill of Maker. GETS TO EXPERT/NOVICE differences!”

In this note, we can see the connection to the codes. Maker design decisions is a code in our codebook. However, rather than simply labeling data and moving on, this researcher wrote about their thinking when coding. They clearly see a pattern between design decisions and the difference between expert and novice participants. This process is ongoing throughout coding and serves as a basis for the meaning-making that transforms data from a list of codes and transcripts into insights and findings that answer our research questions. 

Coding TSTM ETMS data is a rich and iterative process that continues to provide valuable insights into the use of OER, the TSTM instructional toolkit, and the development of the SkillBlox platform. Our structured approach to coding—starting with inductive methods, moving through multiple coding cycles, and culminating in a priori coding based on our codebook— enabled us to categorize and interpret the data. As we move forward, the themes and patterns we are identifying will guide our analysis of the data – supporting our development efforts and ultimately translating the insights gleaned through the qualitative coding process into actionable strategies. Look for more details about these findings in our upcoming blog posts about what we’re learning from our research!