BACKGROUND

Randomized controlled trials (RCT) represent the highest level of research design to establish causal effect of an intervention. However, the ability to establish the effect of a particular intervention depends on the ability to control for external factors that influence internal validity. Assuring internal validity ensures that the independent variable is the only factor that influences change in the dependent variables.[1] When designing an RCT, researchers must consider all possible threats to internal validity, including selection bias, confounding variables, instrumentation, testing, regression to the mean, social interaction, and attrition. Acknowledging the importance of all threats to internal validity within an RCT, the focus of this work is the evaluation of primary and secondary outcomes.[1] 

Several evaluation biases may interfere with our ability to measure causal relationships, such as instrumentation bias. Described as fluctuations in measurement, instrumentation bias can result from variations in inter-rater or inter-rater reliability.[1,2] Inter-rater reliability is defined as the degree of agreement among two or more independent testers, while intra-rater reliability is the degree of agreement among repeated measures performed by a single tester. To limit variations within and between testers, RCT guidelines recommend standardized training of testers to ensure uniform and standardized administration of all outcomes. Such training is particularly recommended in multisite RCT studies.[2]

At present, our team is conducting a 3-site RCT (Quebec, Montreal and Halifax) to evaluate the effectiveness of the Wheelchair Skills Training Program (WSTP) [3] in pediatrics. The WSTP is part of the Wheelchair Skills Program (WSP)[3], an evidence-based program which includes outcome measures and a training guide. The WSP embeds in the World Health Organization (WHO) 8 step process for wheelchair provision, namely step 2 (Assessment) and step 7 (User Training) [4]. In our RCT, wheelchair skills capacity, confidence and satisfaction with participation are being evaluated before and after 12 training sessions and 6 months post-training using the Wheelchair Skills Test (WST) [3], the Wheelchair Confidence Scale for Manual Wheelchair Users, Pediatric version (WheelCon-M-P) [5] and the Wheelchair Outcome Measure for Young People (WhOM YP) [6]. The WST Version 5.3.1 for manual wheelchair users [3] is an objective, performance-based evaluation of manual wheelchair skill capacity comprising 30 indoor, community and advanced skills. Safety in the WST is ensured using a person as a "spotter". The WST provides details of the specific spotting techniques used for each of the 30 wheelchair skills, as well as specific indications for set-up and scoring. The WST has preliminary data demonstrating its validity in the pediatric population. [6] The WheelCon-M-P is a 33-item subjective, self-report outcome measure evaluating manual wheelchair use self-efficacy, or wheelchair confidence. The response scale is a pictorial Likert type scale with descriptors that range from 0 (pictorial: unhappy face; descriptor: not at all confident) to 4 (pictorial: happy face; descriptor: very confident). It has preliminary data for use in children ages 7-18 years with variable diagnoses (e.g., cerebral palsy, spina bifida). [5] The WhOM-YP is a self-report outcome measure testing satisfaction with participation in meaningful activities. Taking the form of a semi-structured interview, the WhOM-YP allows a child (with or without the help of their parent) to create meaningful goals (at home, at school or in the community) with the help of the tester.  During the interview, the PMWUs are asked to rate the perceived 'importance' of the social participation goal (0-10) and 'satisfaction' with current performance of this activity (0-10). Preliminary data shows that the WhOM-YP has been validated for the pediatric population.[6]

Standardized training for the testers of this RCT was particularly important given the challenges faced by our team: 1) varying tester knowledge and skill in administering the outcome measures in pediatrics, 2) lack of training tools specific to the administration of the study outcome measures for pediatric manual wheelchair users (PMWUs); and 3) inherent difficulties related to administering outcome measures in pediatrics. Specifically, the testers involved in the RCT had different experiences and backgrounds, some being research professionals with experience in administering the tools in adults, others being occupational therapists, working in pediatrics, and others having experience in pediatrics and having received WSP training in previous years through either 7-hour continuing education training or through a 13-week Master's level university course. Moreover, available training resources, such as the WST Youtube video series, do not include considerations specific to pediatrics and examples of administration process provide only simulation with adult populations. Finally, conducting an evaluation in a standardized manner while engaging the child is a challenge. In fact, standardized outcome measures like the WST, WheelCon-M-F-P and WhOM YP, frequently have rigid processes and instructions to follow, which are difficult to adhere to, as children and adolescents with disabilities frequently need adaptation and flexibility.[8] For example, they may have difficulty understanding the constructs evaluated (e.g., distinguishing satisfaction and importance in the WhOM-YP) and questions (e.g., figuring what a flat gravel surface is to indicate the confidence level in a maneuvering their wheelchair in the WheelCon-M-F-P), rapidly feel tired and overwhelmed and express discouragement, or simply disengage, when missing several skills at the WST. However, a child-centered approach can facilitate administration (e.g., giving concrete, age-appropriate examples to explain the constructs of confidence and satisfaction).  Therefore, the aim of this study was to co-create a series of instructional videos to enhance internal validity for our multi-site RCT through standardized administration and scoring of all outcome measures. 

APPROACH

We used an iterative co-creation approach to develop a series of instructional videos. Specifically, a PhD candidate and a Master's student, both of whom are occupational therapists with expertise in wheelchair skills testing and pediatric rehabilitation developed the videos. The testers' feedback was incorporated throughout the iterative design process. 

Step1-design

The video series addressed the 3 challenges described above as follows. First, to address varying tester knowledge and skill in administering the outcome measures, the videos were created to fill knowledge gaps for all testers involved in the study, taking into consideration their different backgrounds and experiences.  Second, the creation of the video series addressed the lack of training tools specific to the administration of the study outcome measures for PMWUs. Finally, to address inherent difficulties related to administering outcome measures in pediatrics, the guidelines, and strategies specific to administering the standardized tools to children were included in the videos. These guidelines and strategies were informed by previous research [9], clinical experience, and by the work of the ongoing WSP pediatric sub-committee. The WSP pediatric sub-committee is composed of researchers, clinicians, and wheelchair users from around the world with expertise in wheelchair skills testing and training in pediatric rehabilitation. Thus, the design used for the creation of the video series allowed us to address the 3 challenges faced by our team to ensure standardized testing for the RCT. 

Description of the videos

The creative team developed a series of 6 videos in English and French, varying in length, format and content as described below. The videos were all recorded online through the Zoom platform. Video 1 (5 minute introductory video, narrated PowerPoint presentation) covered considerations in testing wheelchair skills, confidence, and satisfaction in participation in meaningful activities in pediatrics (e.g., establishing a trusting relationship with the child, continuous assent to participate, clarifying parents' roles, using a visual schedule). Video 2 (12 minute video narrated PowerPoint WST presentation) explained specific details on the WST administration (e.g., paraphrasing instructions using a simplified language or gestures, what the testers can and cannot do during the test), the general spotting techniques, pediatric-specific considerations (e.g., how to ensure motivation and engagement in the assessment process without providing feedback on skill performance) and general scoring criteria. Video 3 (4 minute video narrated PowerPoint WheelCon-MP presentation) included a description of the 6-areas that are evaluated, an explanation on how to present the 5-point pictorial likert scale to children and validate their understanding (e.g., rate their confidence level in an activity outside wheelchair mobility), and detailed instructions for the administration process depending on the child's ability to read. Video 4 (16 minute video narrated PowerPoint WhOM-YP presentation) explained the general administration guidelines, the pictorial likert-type scale, and, most importantly, ways of explaining satisfaction in an age and developmentally appropriate manner. Finally, Videos 5 (30 minutes) and 6 (30 minutes) showed examples of WhOM-YP and WheelCon-M-P administration and were intended to provide testers with appropriate and close to realistic models using a simulated administration of the tools. More specifically, in the demonstration of the WhOM-YP, the example focused on guiding a simulated PMWU on developing goals specific to wheelchair skills training. The demonstrations provided a clear and concrete example on the administration process and possible behaviors and reactions to adopt in response to challenges a child may have, without compromising the rigor of standardization.

Step2-video viewing and informal feedback

In this step, the testers were provided online web access to the 6 videos through YouTube for personal viewing as the primary component of the training process. Before receiving the video links, testers were provided with copies of the outcome measures and referred to the WSP manual for self-study. Next, the team solicited the testers and asked questions about their readiness for testing ("Do you feel ready to assess PMWUs?"), unanswered questions ("Is there any incomplete information or unanswered questions in the videos?"), suggestions for improving the training ("What other tools or training would you need to be prepared for testing?"). Finally, the videos were pilot tested with a PMWU and a tester from the Quebec site. This tester did not have any previous experience with the assessments or in pediatrics, prior to the training provided for the RCT. This tester's scoring forms were verified (i.e., review of each skill scored on the WST scoring form, goals developed the WhOM-YP goals, and scoring of the WheelCon-M-P). 

RESOLUTION

The testers considered that the series of 6 instructional videos were clear, coherent, pertinent for their training, and a helpful guide to standardized wheelchair skills testing with the pediatric population, demonstrating that the video series addressed the second and third challenge of our research team, namely the need for training tools and inherent difficulties related to administering outcome measures PMWUs. Feedback received from the testers showed that there was partial readiness to assess PMWUs after viewing the videos (i.e.; information covered in the videos for the WST were general, testers named specifications on spotting, set-up and scoring for each skill would be beneficial) and that they had unanswered questions regarding the WhOM-YP and the WheelCon-MP (e.g.; how many goals are needed, how can we guide PMWUs in developing goals). Testers suggested that the instructional videos were helpful but having in-person training would complete the tester training. 

With regards to the pilot testing, examining scoring of the WST brought about some observations regarding wheelchair components affecting capacity (e.g., adapted transport anchor present under the wheelchair, making getting over obstacles impossible to do). For the WhOM-YP, the objectives developed with the PMWU were skill-oriented rather than function-oriented. WheelCon-M-P scoring was adequate and complete. 

Based on tester comments and reviewing of the scoring forms from the pilot test, it was determined that video viewing alone was insufficient for thorough training since the testers had unanswered questions and did not feel ready to complete the RCT testing with the participants. To address the first challenge faced by the research team (end user [tester] knowledge gap), two in-person educational meetings were added to the training, whereby the testers on each site received 2 additional hours of training especially for answering questions about the WhOM-YP and the WheelCon-M-P and for addressing spotting, scoring and set-up for each of the 30 WST skills. The testers were redirected to the WSP manual [3] to answer pending questions, such as scoring specific skills. The research team was also available to answer questions once the training was complete. Another suggestion from the research team was to film the administration of the WhOM-YP with the participants of the RCT for on-going and collaborative reflection by testers and the research team. Therefore, to ensure rigorous and standardized administration of assessment tools for PMWUs, a wheelchair skills tester package, including a series of 6 instructional videos, a 2-hour in person training, and an open communication channel with feedback given for the semi-structured interview (WhOM-YP), was deemed most effective for training RCT testers. 

IMPLICATIONS

The pediatric wheelchair skills tester package is a product of an iterative and collaborative, co-creation approach developed by researchers and clinicians, addressing the three initial challenges faced by our team in standardizing testing in the WSP-Ped RCT. The tester package is a promising tool for training testers in administering and scoring standardized outcome measures for testing wheelchair skills, confidence and satisfaction in participation in meaningful activities through validated outcome measures namely the WST, the WhOM-YP and the WheelCon-MP, enhancing inter-rater and intra-rater reliability in our multisite WSP-Ped RCT. The research team proposed future research perhaps through focus group discussions involving knowledge users for continuous improvement and tailoring for standardized training and, possibly for clinician training. More specifically, the package is an interesting addition to the already existing instruction videos available on the WSP Youtube channel. Training clinicians with the tester package will promote best practice in evaluating wheeled mobility, as recommended in Step 2-assessment of the WHO's 8 step process. Dissemination of the pediatric wheelchair skills tester package can be done locally (i.e.; through online organizational platforms) and through the WSP YouTube channel (WST Youtube video series) for widespread use. Disseminating the tester package will contribute to the research team's recent efforts to implement the WSP in pediatric settings.