RESNA 26th International Annual Confence
ENSURING EFFECTIVE TECHNOLOGY TRANSFER THROUGH EVALUATION
This paper presents partial findings from a research and development program that is ongoing at the University at Buffalo's Center for Assistive Technology. The research component of this program validates an innovative approach to Technology Transfer as applied to the field of Assistive Technology. This paper focuses on the program's development component and describes the role of evaluation in the methods used to transfer technologies and introduce products into the marketplace that address high priority needs of persons with disabilities.
The Rehabilitation Engineering Research Center on Technology Transfer (T2 RERC) seeks to improve the quality of life for persons with disabilities, by transferring needed technologies and products into the Assistive Technology (A/T) market. Funded by the National Institute on Disability and Rehabilitation Research, the program has completed four years of its five-year funding cycle. The T2 RERC uses a comprehensive and systematic approach to accomplish its transfer goals and studies the proposed model as it implements it. (1). The program's research efforts on evaluating the model are discussed elsewhere (2). This paper will describe the evaluation methods and findings from the program's development projects and how the methods facilitate the transfer of the targeted technologies and devices to the marketplace. While the transfer process basically involves transforming an idea into a commercialized product through a prototype, two development projects operate in the program, and reflect the two market forces that drive the transfer process. In the Demand Pull (DP) project, consumer need and market potential are established prior to finding technology solutions to address these needs and in the Supply Push project (SP), candidate technology solutions are identified prior to validating consumer needs and market potential and facilitating the adoption of these solutions by the A\T industry. Both projects involve contributions of all of the stakeholder groups critical to technology transfer effectiveness: technology developers, manufacturers adopting solutions and making products, product consumers and resource providers (funding, public policy, etc.) Operationally speaking, protocols of both projects accomplish the transfer in a stepwise fashion, producing sequential outputs. (See Fig. 1). Each output carries new knowledge generated by preceding research. This knowledge represents an intermediate outcome, which feeds into the next step and its outcome, leading to the final step of transfer. The outputs are thus "carriers" of the transfer process as they "carry" the new knowledge forward.
| Technology "A" is lacking in A\T Mkt. |
→ |
Evaluative (Knowledge generating) activities |
→ |
Outcome One (Carrier One) |
→ |
Evaluative (Knowledge generating) activities |
→ |
Outcome Two (Carrier Two) |
→ |
Technology "A" enters A\T Mkt. |
|
Evaluation ensures both the quality and relevance of the DP and SP outcomes. In its formative role it improves the quality of the outcomes while they are in process. (3). By pro-actively judging the quality of the intermediate outcomes and improving the process as needed, we achieve the final outcome in the desired quality. We thus optimize the quality of both the intermediate outcomes and the processes. In its summative role, evaluation next verifies if in fact the desired quality is achieved in the final outcome. In addition to quality, evaluation ensures the relevance of the outcomes through a context evaluation done prior to the in-process outcome evaluations. (4). Both DP and SP projects conduct stakeholder needs assessments prior to outcome development in order to ensure that the outcomes will in fact be relevant to the expressed consumer needs.
For summative evaluation, the overall indicator of a transfer accomplished (or not accomplished) is a "product licensed to market" or an equivalent event. Formative evaluation focuses on the specific intermediate outcomes (and the carriers), which are different for DP and SP, within the sequence of operations in Fig.1. In principle, the indicator of an effective outcome is the quality and quantity of the next outcome that depends on it. However, separate feedback on the outcomes gives additional assurance of quality. Under the DP process (5), primary market research compiles an Industry Profile and White Paper that describe the relevant industry and technology needs. These intermediate outcomes form the basis for moderated discussions at a stakeholder forum whose output is used to produce Problem Statements that outline desired technology solutions and their market potential. The statements are then used to locate technology solutions through a variety of outreach activities to technology developers. Under the SP process, new and unique inventions are solicited and invited into the program. These inventions then undergo a rigorous set of consumer, technical and marketing evaluations for entering into the T2 RERC's commercialization process. The SP team then compiles a commercialization package based on findings and reports. The package includes: evaluation findings, intellectual property, background information and relevant appendices as well as an executive summary.
Meeting stakeholder needs is a criterion par excellence of the quality of our final outcomes. So our critical data source is primary market research using pertinent stakeholders, complemented by secondary market and technical research. Besides informal stakeholder feedback, our formal methods of observation include: focus groups, consumer panels and surveys used in SP; and phone interviews with experts, end-user panels, formal surveys and checklists used onsite at our stakeholder forums, for DP. We also keep ongoing logs of practices modified in function of formative evaluation and organize them in one-on-one interviews with the DP and SP managers. Our sampling is judgmental. Heterogeneous end-user populations, multiple device features, practical restrictions of cost and logistics limit our sample size. Directing and supporting quality outcomes is of more concern than generalizing findings. We include "information-rich" cases in purposeful stratified samples and achieve optimum variation through snowball recruiting of persons with multiple expertise. For example, we achieved representation of 74 distinct areas of knowledge on features of an AAC device for various disability needs by recruiting 41 people purposively.
To date the SP project has achieved 13 licenses and DP has accomplished 6 transfers in major technology areas. Ensuring the quality of the intermediate outcomes was important to their success. Our findings from SP focus groups and surveys on purchase intent, cost estimate and design requirements were varied and device specific, and very useful for our decisions to commercialize or not. Details are omitted here for reasons of space. Cost and sales volume estimates of several devices closely approximated those for other products in that market sector, increasing the confidence in our focus group methods. For the DP project, the white papers evaluated by the stakeholder forum surveys showed high acceptance levels - mean of 4.2 on a 5 point scale for wheeled mobility (n=46); mean of 4.5 for hearing enhancement (n=45) and mean of 4.5 for communication enhancement (n=45). Each forum was praised by all stakeholder groups. Subsequent informal feedback about the forum proceedings revealed that our stakeholders value them. Our effort to disseminate problems and locate proposals was not formally evaluated by stakeholders, but for now an indicator of their quality is the number of proposals they obtained - 38 over a period of 3 years for wheeled mobility, 19 over 2 years for hearing enhancement and 7 over a one year period for communication enhancement. Relevance of outcomes to stakeholders is also an indicator of quality, which has been evident in their continuous acceptance of the program through feedback. They are satisfied with our SP commercialization packages. They have consistently expressed appreciation of our DP forums - as opportunity to interact with and learn about expertise of other stakeholders, and to network with similar groups. A leading manufacturer in home appliances and then a Fortune 200 company have initiated partnership with us in taking their product successfully to market. Tracking users of our website shows industry and business as a major category (about 40% have .com extensions) of interested groups.
The learning from formative evaluation brought about several changes in our best practices built into the model. We dropped an SP carrier "Agent agreement" that proved ineffective. The standard set of carriers did not apply in total for some SP devices, and we developed alternate paths to market (6). Refinements to DP carriers have occurred through a steady protocol evolution over the four years. Improved practices include: guidelines and rationale for the use of purposive sampling, effective recruitment tactics, procedures for focus group moderation for SP and DP, formats and strategies ensuring "full inclusion" of consumers in mixed focus groups of persons with and without disabilities; alternative surveying procedures for primary market research such as the Kano approach; diversification of transfer strategies far beyond simple technology licensing; and diversification of outreach activities far beyond simple shotgun dissemination (e.g., news articles).
This is a publication of the Rehabilitation Engineering Research Center on Technology Transfer, which is funded by the National Institute on Disability and Rehabilitation Research of the Department of Education under grant number H133E980024. The opinions contained in this publication are those of the grantee and do not necessarily reflect those of the Department of Education.
Dr. Vathsala I. Stone
Research Director,
RERC on Technology Transfer,
610 Kimball Tower,
University at Buffalo,
Buffalo, NY 14214;
716/829-3141;
vstone@acsu.buffalo.edu