Vathsala Stone1 and Machiko R. Tomita, 1,2
Center on KT4TT,1 Department of Rehabilitation Science,2 University at Buffalo
INTRODUCTION
There is a growing concern about obtaining beneficial social impact from research and development (R&D) projects sponsored through public funding. Knowledge Translation (KT), which responds to this concern, upholds knowledge utilization as a desired research outcome. In the specific case of technology based R&D, technology transfer (TT) processes form an integral part of KT best practices. The KT4TT Center is funded by the National Institute on Disability and Rehabilitation Research (NIDRR) at the University at Buffalo for developing such KT best practices. The center is conducting a series of randomized controlled studies to evaluate effectiveness of KT interventions focused on NIDRR funded R&D projects. The first RCT in the series addressed Augmentative and Alternative Communication technology. This presentation focuses on the key results.
Purpose of the Study
The purpose of the study was to evaluate the effectiveness of three methods of KT intervention in terms of raising overall levels of knowledge (K) use by stakeholders in the field of Augmentative and Alternative Communication (AAC) technology. The three methods are tailored and targeted dissemination of knowledge (TTDK), targeted dissemination of knowledge (TDK), and passive diffusion.
Research Question (RQ)
RQ1. Do knowledge use levels change over the study period for the three groups?
RQ2. Are changes in knowledge use levels over the study period different among the three groups?
RQ3. How many people move from non-awareness level to awareness level and higher in each of the three groups?
This step was particularly important in the study because if stakeholders did not know the innovation, they would not use it. Raising awareness was the first step to facilitate its use.
RQ4. How many people move from non-use level to use levels in each of the three groups?
METHODS
The Interventions
The Age Appropriate Vocabulary and Symbols Set conceptualized for adult users of AAC by Bryen (2008) was selected as the focus of the intervention, based on the criteria of utility, feasibility and innovativeness. Five types of stakeholders participated in the study: manufacturers, brokers, clinicians, consumers and researchers. They were recruited through their organizations of affiliation, which were pre-profiled on the basis of organizational value- mapping (Lane & Rogers, 2010). Two of the three communication strategies focused by the intervention were TTDK and TDK. They were both compared to passive diffusion, used as Control in this study. Changes in levels/types of knowledge use over time formed the dependent variable, measured by Level of Knowledge Use Survey (LOKUS), specifically developed for the study.
For the TTDK method, stakeholder groups were “targeted” as audience via relevant professional organizations; the innovation (Bryen’s AAC vocabulary list) was “tailored’ to the specific contexts of these groups and delivered in multiple modes. Participants received Bryen’s research article, with a Contextualized Knowledge Package (CKP) followed by a tailored webcast and offer of technical assistance.
The CKPs were produced one for each stakeholder category using the language and format appropriate to each stakeholder. Each CKP introduced and presented Bryen’s innovation, summarizing the research, describing the innovation and its potential benefits to the specific stakeholder, explaining its use in their specific professional/personal contexts and its potential benefits. The webcasts used a central video that demonstrated the use of the new vocabulary with an AAC device, integrating this with different narratives addressing different stakeholder groups, separately.
The TDK method consisted of dissemination of Bryen’s research article to “targeted” stakeholder groups, but without any tailored material. In the study, both methods were compared to the traditionally practiced method by NIDRR grantees, or passive method of diffusion serving as a control group.
Research Design
Figure 1: Research Design dWe employed a randomized pre- and posttests design using three between-groups. Participants responded to LOKUS three times: baseline (pretest), four months (follow- up 1), and eight months (follow-up 2). After the baseline online survey, the TTDK group received Bryen’s research article, and a stakeholder-relevant CKP. The TDK group received the original article only; and the control group did not receive any intervention. During the second 4-month period, the TTDK group further received a tailored webcast on the same innovation. Online survey was administered by sending a link to the survey three times to each participant. Figure 1 summarizes the study design.
Sample
Sample size was determined by power analysis based on a prior study in literature. We needed 206 participants to achieve power of .80 at α2=.05 with a small effect size. Considering 24 % attrition, a total of 270 participants were decided to be recruited. The inclusion criteria were aged 18 years or over and a member of specific stakeholder organizations. For the recruitment of participants, the announcement of the study was made thorough each stakeholder’s organization. Individuals who were interested in this study contacted the Center for KT4TT. Then after verifying their eligibility to this study, they were assigned to be one of the three groups randomly.
Instrument
The LOKUS was the instrument used to identify the effectiveness of the intervention in this study. It consists of four levels of knowledge use (See Table 1). The first three levels are sequential including non-awareness, awareness, and interest. The last level consists of intended use and modified use, which are two choices. Levels 3 and 4 contain two to four dimensions. LOKUS is an online survey, and it takes 15 to 20 minutes to complete. Its psychometrics had been established (Tomita et al, in preparation). This online survey used Vovici 6.0 survey platform.
| Levels |
Dimensions |
|---|---|
1. Non-awareness |
|
2. Awareness |
|
3. Interest |
Orientation, Preparation |
4. Intended use |
Initial use, Routine use |
4. Modified use |
Collaboration, Expansion, Integration Modification |
Analytical Scheme
Since the dependent variables were measured using an ordinal scale, all statistical analyses used nonparametric statistics. For RQ1, change in the level of knowledge use within a group was measured using Friedman Test. For significant results, post-hoc tests were conducted using three Wilcoxon Signed Ranks Tests with Bonferroni correction. For RQ2, three groups were compared for the changes from baseline to F-up 1, baseline to F-up 2, and F-up 1 to F-up 2 using Kruskal-Wallis one-way ANOVA for independent samples. The post-hoc tests were conducted using a minimum significant difference. For RQ3 and RQ4, to analyze frequency of change from non-awareness level to higher levels (that is, awareness, interest and use combined), and from non-use levels (non-awareness, awareness and interest combined) to use levels, all groups were collapsed into 2 x 2 table, and the McNemar Tests for correlated samples were used.
For all analyses, SPSS 18.0 was used and the significance level was set at .05 for primary analyses.
RESULTS
Participant Characteristics
A total of 207 participated, of which 72, 72, and 63 were in the TTDK (T1), TDK (T2), and Control groups (C), respectively. All three groups were equivalent for demographic characteristics including age, years of experience, gender, race/ethnicity), education level, and work status (p>.05 for all).
RQ1. Change in knowledge use levels over the study period for the three groups
Both TTDK and TDK moved up levels significantly from baseline to F-up 1 and Baseline to F-up 2 but not from F-up 1 to F-up 2. The Control group also moved up, but without statistical significance. Table 2 shows means and standard deviations of knowledge use levels at three points of time for the three groups.
T1 (TTDK) n=72 |
T2 (TDK) n=72 |
Control n=63 |
|
|---|---|---|---|
| Baseline Mean (S.D.) | 1.22 (.68) | 1.26 (.77) | 1.38 (.97) |
| F/up 1 Mean (S.D.) | 1.79 (1.16) | 1.76 (1.19) | 1.51 (1.05) |
| F/up 2 Mean (S.D.) | 1.69 (1.03) | 1.74 (1.16) | 1.73 (1.22) |
| Difference χ2 (p) | 22.632 (<.001) | 13.884 (.001) | 6.484 (.039) |
| Post-hoc test Z (p) | Base.vs F-up1 3.826 (<.001) | Base.vs F-up1 3.330 (.001) | |
| Base vs F-up2 4.297 (<.001) | Base vs F-up2 3.206 (.001) |
RQ2. Change in knowledge use levels over the study period; difference among the three groups
As Table 3 summarizes, statistical significance was found only for the changes occurred from baseline to F-up1. The changes of knowledge use level were different for T1 (TTDK) vs. Control and T2 vs. Control, but not for T1 and T2.
| Knowledge Use Level Change | T1 (TTDK) Mean (S.D.) | T2 (TDK) Mean (S.D.) | Control Mean (S.D.) | Diff. χ2 (p) |
|---|---|---|---|---|
Baseline to F-up 1 |
.57 (1.12) | .50 (1.17) | .13 (1.01) | 7.044 (.030) T1,T2vs.C |
| Baseline to F-up 2 | .47 (.82) | .47 (1.19) | .35 (1.19) | 2.371 (.306) |
| F-up 1 to F-up 2 | -.10 (1.20) | -.03 (.75) | .22 (1.13) | 3.443 (.179) |
RQ3. Change in knowledge use levels from baseline to F- up1: Non-Awareness to Awareness and above
For the TTDK group, 63 participants were in the Non- Awareness level at baseline and 30.2% moved up to higher levels at F-up1 with significance (p<.001). For TDK, very similar pattern was observed. There were 63 participants, who were not aware of the study at baseline, and 27.0% of them moved up levels at F-up1. This change was significant at <.001. For the control group, 54 participants were not aware of the study, and 88.9% remained unchanged at F- up1. Table 4 shows the information for the three groups.
| Follow-up 1 | |||||
|---|---|---|---|---|---|
| Baseline | Non- Awareness |
Awareness and above | Total | p | |
| TTDK | Non- Awareness |
44 (69.8%) |
19 (30.2%) | 63 | <.001 |
| TDK | Non- Awareness |
46 (73.0%) |
17 (27.0%) | 63 | .001 |
| Control | Non- Awareness |
48 (88.9%) | 6 (11.1%) | 54 | .289 |
RQ4. Change in knowledge use levels from baseline to F- up1: from non-use to use level
Analyses were performed on the data change between baseline and F-up1, comparing frequencies corresponding to “non-use” level (that is, non-awareness, awareness and interest combined) and to “use” (that is intended and modified use combined) level. Table 5 summarizes the results for participants who were in the Non-Use level for the three groups.
| Follow-up 1 | |||||
|---|---|---|---|---|---|
| Baseline | Non- Use |
Use levels | Total | p | |
| TTDK | Non- Use |
59 (85.5%) | 10 (14.5%) | 69 | .039 |
| TDK | Non- Use |
57 (83.8%) |
11 (16.2%) | 72 | .022 |
| Control | Non- Use |
52 (92.9%) | 4 (7.1%) | 63 | 1.00 |
As shown in Table 5, 69 participants the TTDK group were in the Non-Use level at baseline and 10 of them (14.5%) moved up to the levels with statistical significance (p=.039). For the TDK, a similar pattern was observed and 16.2% of participants who were in the Non-Use level, moved up to Use level with statistical significance (p=.022). The control group change was not significant.
DISCUSSION
This study evaluated the effect of three KT strategies on the level of knowledge use generated by technology based R&D projects with focus on an innovation in AAC technology. The essence of TTDK was audience-targeting and tailoring or contextualization of knowledge, both part of the Knowledge-to-Action (KTA) model proposed by Graham and colleagues (2006). This model considers the generated knowledge as a solution and, in taking it to action, a “match” must first be found in terms of problems that it can solve. In other words, relevant users should be targeted. The model also recommends tailoring or putting the knowledge in the context of the users.
This study showed targeting effective, which underscores the importance of “relevance” as a factor that bridges knowledge to potential users. However, tailoring was not supported as an effect enhancer, thus favoring TDK and questioning the value of additional cost of tailoring. Conclusions are tentative pending future replication studies.
The CKP was effective but not webcast. This raises further questions as possible factors, such as the order of administration, the duration of the study period, among others. Replication studies may consider variations in this regard.
Qualitative results in terms of proportions suggested the relative ease of raising awareness vis-à-vis influencing the decision of non-users to use. As the old adage goes, “making the horse to drink” as opposed to “leading him to water” is a challenge. It has long been recognized as a problem in innovation diffusion literature (Rogers, 1962). Several factors, contextual or personal barriers and opportunities may explain the gap between interest and use. Importantly, however, pre-existing user need is a strong motivator for using an innovation. The results are hardly surprising, if we consider that both end-of-grant and integrated KT concepts in the KTA model are “knowledge- push” approaches where knowledge is generated first with an assumption that the corresponding user need exists. On the other hand, prior-to-grant KT approaches (Lane & Flagg, 2011) recommend that generating knowledge should be undertaken in function of a pre-identified and validated need. Whether this approach might be effective in taking the interested user to the use level is an interesting question for future RCTs.
CONCLUSIONS
The study showed TTDK and TDK interventions are effective as strategies for disseminating new knowledge generated by the chosen innovation in AAC. Both strategies target specific audiences, and they were effective in increasing awareness and use levels in stakeholders. However, tailoring, which was the additional component in the TTDK intervention may not have additional effectiveness. Further studies are necessary to conclude this.
KEY REFERENCES
Bryen, D.N. (2008). Vocabulary to Support Socially- Valued Adult Roles. Augmentative and alternative Communication, Vol. 24, No. 4, Pages 294-301
Graham, I.D., Logan, J., Harrison, M.,B., Straus, S.E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in translation: time for a map? The Journal of Continuing Education in the Health Professions, 26(1), 13-24.
Hall, G.E., Dirksen, D.J., and George, A.A. (2006). Measuring Implementation in Schools: Levels of Use. Austin, TX: Southwest Educational Development Laboratory (SEDL).
Lane, J.P. and Rogers, J.D. (2011). Engaging national organizations for knowledge translation: comparative case studies in knowledge value mapping, Implementation Science, 6:106.
Lane, J.P. and Flagg, J. L. (2010). Translating three states of knowledge--discovery, invention, and innovation, Implementation Science, 5:9.
Rogers, E.M. (1962). Diffusion of Innovations. Glencoe: Free Press.
ACKNOWLEDGEMENTS
This is a presentation of the KT4TT Center which is funded by the National Institute on Disability and Rehabilitation Research of the U.S. Department of Education, under grant number H133A080050. The opinions contained in this presentation are those of the grantee and do not necessarily reflect those of the U.S. Department of Education. We also acknowledge collaboration and expert input from the RERC on Communication Enhancement during the implementation phase of the study.