3.1. Data collection

Data for this study were collected in the spring of 2024 in an undergraduate product design and innovation course at a large, public R1 university in the United States (Reference Codner and LauffVang & Lauff, 2024). Consent and voluntary participation in this study were obtained beforehand and approved by the Institutional Review Board (IRB) under STUDY00021116 from students and stakeholders. Ethical guidelines were followed to ensure the safety of children included in the study. No additional coursework was needed from students who opted in and participation in the study had no impact on their course performance. This course was intentionally structured to include students (n=62) from various backgrounds (i.e., mechanical engineering, product design, marketing, computer science) to engage in a client-sponsored project, focused on toy innovation. Ten design teams were formed with six to seven members per team based on skills and experiences to develop diverse teams.

This study focuses on five design teams (six students/team), identified as fruit pseudo-codes for each team: raspberry, cherry, apricot, grape, and lemon. We chose these five teams to analyze because of their consistent stakeholder group testing (child/parent and industry experts) and higher-quality video recordings without major audio/video issues. The other five design teams from the class were excluded for data consistency, due to the following reasons: (a) testing with different stakeholder groups due to the direction of their concepts (i.e. designing for pets instead of children), (b) major audio issues that interfered with the quality of the data collected, and (c) failed initial video recording during prototype testing sessions. Throughout the 16 weeks, each team was tasked with a series of prototype checkpoints or design iterations. Students started with concept sketches before moving into prototyping activities for the remainder of the semester. Then, each design team created six “sketch models” or low-fidelity prototypes in the first checkpoint (CP1). Next, the teams refined and iterated the six sketch models into two “works-like” medium-fidelity prototypes in the second checkpoint (CP2). In the final checkpoint (CP3), each design team created one final “looks-like” and “works-like” model (see Figure 1).

Figure 1. Prototype evolution of five design teams across three design checkpoints

3.3.1. Prototype testing sessions

Student design teams conducted prototype testing sessions with key stakeholders to inform future design directions at each checkpoint. A short lecture on best practices and testing guidelines was given in class as an introduction to prototype testing to baseline knowledge; this lecture was a typical part of the course and typically takes about 30 minutes. For example, best practices such as reminding participants that teams are not testing them, starting with broad questions before getting specific, and encouraging think aloud during prototype interactions were shared. Additionally, every team received note-taking guides with testing tips to help structure the sessions and note any observations. Before each testing session, time in class was provided to plan and develop testing approaches (i.e., time management, member roles, goals, and testing objectives). Two volunteer industry lab instructors were assigned to each design team and assisted by sharing their industry knowledge about prototype testing throughout the course. Despite these preparations, the testing approach was left to each team to decide how they structured and conducted the testing sessions since each team had different prototypes and feedback goals for testing. Historically, this portion of the design process is less structured in what to prototype and how to test, which presented a unique opportunity to investigate how student designers approach prototype testing sessions with stakeholders.

The research team organized and recorded the prototype testing sessions during Thursday night lab periods from 6-9pm. Each team had one hour of prototype testing time for each checkpoint: testing with an industry expert (E) for 25 minutes and testing with child/parent (C/P) for 25 minutes with a short transition between testing sessions (see Figure 2). The research team informed the design teams about the video recordings, emphasizing that students were not evaluated on their performance but rather curious to learn and understand how they approach prototype testing. To alleviate any pressure or biases of impacted behaviors, the course professor was removed from the testing environment and results were not shared until after the course ended. For this paper, 30 video recordings were collected and analyzed from five design teams. These 30 prototype testing sessions equate to about 10 hours of active prototype testing with 22 stakeholders and with 45 prototypes (each team had 6 low-fidelity models at CP1; 2 medium-fidelity models at CP2; 1 high-fidelity model at CP3).

Figure 2. Prototype Testing Set Up

3.2. Data analysis

The data analysis used a deductive and exploratory approach based on an a priori coding framework. The video recordings were analyzed using NVivo, a qualitative data analysis software. NVivo has tools to “code” timeframes of the recording and organize the video sections according to specific themes. Coding is the process of tagging parts of the data to highlight themes and interconnections (Reference Braun and ClarkeBraun & Clarke, 2012). This coding process was completed by the lead research assistant in recurring meetings with the primary investigator to review the codes. The coding framework came from a study of novices’ early-stage design interviewing skills (Reference Grover, Wright, Hoody and LauffGrover et al., 2022). Due to the nature of early prototype testing sessions acting as “probes” to help the teams identify new information through asking questions and encouraging shared thoughts, we chose this framework to code the prototype testing sessions. The research team engaged in an initial trial coding of the data using three prototype testing videos to ensure that this framework would apply appropriately to this new data. After this trial, the original framework was slightly adapted to focus on prototype testing interactions based on our learnings (see Table 1).

Table 1. Adapted codebook from interviews to prototype testing sessions (Reference Grover, Wright, Hoody and LauffGrover et al., 2022).

The videos were coded for the designers’ interactions only, as the focus was on understanding their approaches, not the stakeholders (i.e., parents). The codes in NVivo are for a point in time (instance) when the team used a specific type of testing interaction, and do not include the duration of the use. We coded all verbal statements/questions from the designers, along with non-verbal interactions that included gestures and head nods when related to the established code (i.e., head nods as a type of active listening). Additionally, all questions were double coded to the types of questions and specific contexts (i.e., follow-up). For example, a direct question based on stakeholders’ actions or words was coded as “Follow-Up Questions,” with “Open-Ended Questioning” applied if the question was open-ended. An expert in prototyping research reviewed the coding process and results, leading to two iterations and a final review. Figure 3 illustrates the coding process with interaction codes divided into the opening, body, and closing sections for one prototype testing session.

Figure 3. Example: testing interaction codes in NVivo for one prototype testing session

5.1. Design team dynamic x testing interactions

Results reported in Section 4.1 could be interpreted as whether the testing interaction occurred or not, and how many times per testing session. Since each design team has six total prototype testing sessions, there should be six instances of “Building Rapport” and “Closing Remark,” ensuring that stakeholder expectations are aligned. However, we observed that 16 prototype testing sessions dove straight into testing. This could be due to (a) the pressure to get started and finish within 25 minutes, (b) the lack of experience interacting with stakeholders to build rapport, and/or (c) the need to adapt each testing session to stakeholders. The teams spent time setting up the testing sessions, but they did not always “properly” conclude (Reference Grover, Wright, Hoody and LauffCardoso et al., 2020; Reference Cardoso, Hurst and NespoliGrover et al., 2022). Despite the high instances of “Team Support,” there seem to be challenges with the team dynamics where the teams rely less on member support to focus more on creating a comfortable environment for stakeholders. For example, Team Cherry had one moderator and two notetakers while Team Raspberry’s members had less defined roles. “Signposting” includes time management, but none of the teams had a dedicated member to track time, leading to a rushed closing of the session or complete dismissal of closing remarks. Since each team has six members, defining roles for all members is crucial to ensure full engagement. Undefined roles for some may leave team members idle, impacting the structure and approach to prototype testing.

5.2. Interesting observations among different stakeholders

The results in Section 4.2 highlight how the design teams structure the body section (typically 15-20 minutes) of the prototype testing sessions. There are noticeable differences in the testing approach for the two stakeholder groups, which should be explored with further research. Children aged 4-9 participated in the first stakeholder group with parents. Younger children (ages 4-6) struggled with attention and answering questions (Reference Donker and ReitsmaDonker & Reitsma, 2004), which may explain the higher frequency of “Closed-Ended Questioning” with children compared to experts. For example, asking, “Do you like this color?” to a 5-year-old child can elicit either a head nod or a yes/no response. There is value in short, direct responses when closed-ended questions are used appropriately. When the child was older (ages 7-9) and could form a dialogue with the design team, more open-ended questions were used. Open-ended questions such as “What are you feeling? What do you think?” enable older children to share their thoughts and have parents less involved. This shows that closed-ended questions can help design teams gather more feedback from children without negatively affecting quality (Reference Mohedas, Daly, Loweth, Huynh, Cravens and SienkoGrove et al., 2022; Reference Grover, Wright, Hoody and LauffMohedas et al., 2022). However, the design teams treated prototype testing with industry experts as consultations, where experts led the discussion. This informal approach fostered collaboration and may explain the “Leading/Biased” instances observed in all teams. Experts tend to ask more questions upfront to understand the prototypes, elucidating why “Signposting” and “Building Rapport” instances are lower. “Encourage Think Aloud” also appeared differently for each stakeholder group. For example, children expressed their thoughts through nonverbal and verbal cues, like repeated touches and comments such as “This is cool.” This required the design teams to be more intentional in encouraging shared thoughts, using observations as the primary method while soliciting thoughts from experts had a greater emphasis on verbal feedback (Reference Grover, Wright, Hoody and LauffDonker & Reitsma, 2004; Reference Donker and ReitsmaBanker & Lauff, 2022). Despite these differences, the teams asked new questions more frequently rather than follow-ups or reflective questions. Possible explanations include (a) having pre-set questions, (b) disorganized roles disrupting the flow, and/or (c) prioritizing quantity of questions over quality and depth of questions.