INTRODUCTION

The National Institute on Disability Independent Living and Rehabilitation Research (NIDILRR) sponsors dozens of research projects and development projects, all of which are required to generate outputs that will be transformed into outcomes that will have beneficial impacts on key stakeholders (i.e., Persons with Disabilities, family members or care providers).  NIDILRR has separately defined Stages of Research and Stages of Development to differentiate the specific outputs and outcomes expected from these two types of projects [1]. 

Research projects have a long-term orientation.  Investigators apply scientific methods to generate new knowledge in the form of conceptual discoveries.  The path from project implementation, through outputs and outcomes, and on to impacts is inherently indirect and diffuse.   It is challenging to even track the progress of research projects over time, so stakeholders accept the assumption that scientific research eventually yields useful outcomes and beneficial impacts.

In contrast, expectations for outputs from development projects are shorter-term.  Development projects apply engineering methods and/or clinical interventions to generate new knowledge in the form of tangible prototypes that are intended for application in practice by clinicians, manufacturers, policymakers or consumers [2].  The paths to outcomes and impacts are more direct, involving targeted partners and stakeholders.  Therefore, tracking project progress to tangible outputs (i.e., draft documents, functional prototypes/applications), and then on to transfer outcomes as they occur certainly seems feasible in practice.  Doing so provides Grantees and the sponsor with a unique opportunity to demonstrate results achieved during or soon after project completion.

A prior retrospective by the authors of this paper (study team) analyzed the results demonstrated by a total of seventy-eight development projects initiated in 1998/1999 by NIDILRR-funded Rehabilitation Engineering Research Centers (RERC’s).  Only projects that expressly proposed to achieve transfer outcomes were included in the analysis [3]. Findings showed a high level of success for projects generating standards/guidelines, in contrast to a low yield of prototype outputs (< 50%), and transfer outcomes (15% - 30%) for projects intending to generate instruments/tools, freeware or commercial products.

NIDILRR cited these findings in a 2008 competition to establish the Center on Knowledge Translation for Technology Transfer (KT4TT), charged with improving overall Grantee performance when conducting development projects.  The study team secured that funding to create a suite of evidence-based models, methods and metrics, tailored to Grantee’s, and sufficient to plan, implement and complete such projects [4]. 

The study team secured a subsequent round of NIDILRR funding (2013-2018) to further tailor that suite of tools and resources to the expressed needs of on-going Grantees.  The same team concurrently initiated a prospective study of all development projects initiated by NIDILRR Grantees in the 2013/2014 funding cycles to study Grantee performance over time and to better understand actions and decisions as they occured.  This paper summarizes the prospective study’s method, provides a preliminary overview of Grantee performance during the study’s timeframe, and describes key factors contributing to results demonstrated up to the study’s 2018 conclusion..

METHODS

The prospective study design enabled the collection of data as investigators implemented and conducted projects expressly intending to generate any of four types of technology-oriented outputs:  1) Standard/Protocol; 2) Freeware; 3) Instrument/Tool; 4) Commercial Product.  Following Grantee projects over time permitted the study team to identify actions and decisions that either facilitated progress (carriers) or impeded progress (barriers).  The design also enabled comparisons to established industry best practices and the identification of ad hoc approaches that might qualify as best practices within the unique constraints of the AT field. 

All NIDILRR Grantees who initiated technology-oriented development projects in either 2013 or 2014 – identified through their project descriptions submitted to NARIC -- were invited to participate. Forty projects proposed by twenty-two Grantees from different programs (Ten RERC’s; Five SBIR Phase II’s; Six Field-Initiated Development; One DRRP) met the study’s inclusion criteria.  Of these, thirteen Grantees representing twenty-two projects volunteered to participate (Group A), while nine Grantees representing twenty projects declined (Group B).  The study team followed each project separately as individual cases, even though some Grantees proposed multiple projects.  The study design used two different approaches to follow projects in Groups A and B.

For the twenty projects in Group A, individual investigators first shared their funded proposal narratives, and then participated in quarterly interviews with two members of the study team.  The interviewers took detailed notes about actions taken and decisions made during the prior quarter, highlighting progress made and offering technical assistance where appropriate.  All interviewees except one gave prior consent to recording their interviews for accuracy.  The same team members transcribed interview recordings within NVivo software then independently verified the accuracy of content.  The completed transcripts were then analyzed along with questionnaires completed by interviewees through NVivo, which is designed to extract and organize information from qualitative and mixed-methods data.  Analysis revealed that three Group A projects intended two outputs each, bringing the total number of outputs tracked to forty-two.

For the twenty projects in Group B, the study team followed their progress by obtaining their original funded proposals and their subsequent annual performance reports (APRs) from the sponsor agency, via Freedom of Information Act (FOIA) requests.  This approach ensured that the study team received the official record of progress reported to NIDILRR.  Two members of the study team not involved in Group A interviews independently extracted and verified project details from these official records for all twenty Group B projects.

All information collected through primary sources for both Group A and Group B was supplemented with evidence drawn from secondary sources (e.g., presentations, papers, app store metrics, websites) regarding progress towards prototype output, transfer outcome and possibly stakeholder use.  While the study team continues its detailed analysis for the forty projects, sufficient work is completed to summarize and report preliminary findings about key facilitators and barriers to progress shared by multiple projects.

Limitations: First, due to no-cost extensions as well as supplemental funding, some on-going projects may yet demonstrate results that are not yet evident.  The study team was unable to secure supplemental funding to follow all projects to their conclusion.  Second, because the Group B cases lack quarterly interview data, the results for all forty projects are directly comparable only in terms of their achievement of output or outcome results. The ongoing analysis of actions taken and decisions made by Group A investigators will lead to future reports.

RESULTS

The results achieved by Group A and Group B projects, as well as evidence of progress toward those results, provides an overview of the Grantees’ performance on development projects intending to achieve technology transfer outcomes (Table 1).  As noted, these results are provisional for projects still in process.  However, progress to date is a strong indicator of likely future achievements especially for projects exhibiting little activity.

Three-fourths of all proposed projects have demonstrated a functional prototype within the study’s period.  The two or three year projects that were entirely focused on delivering one single outcome (i.e., SBIR Phase II and FI-D) had a higher overall rate of achievement, than did the five-year RERC projects, where investigators leading development projects were likely also involved in research related projects.  As found in the prior retrospective study [3], investigators and graduate students typically prioritize the research projects which are critical for generating publications necessary for academic career advancement.  Further, Grantees are acutely aware that research activity is preeminent in all of NIDILRR’s performance metrics, so they focus efforts on research.

Overall, the current cohort of projects delivered a slightly greater percentage of transfers across all four types of output, than did the cohort of projects initiated fifteen years ago.  Data shows that much of the improvement is attributable to the fact that two-thirds of the current projects had proposed new software (freeware or commercial applications) as their outcomes.  Grantees can now achieve evidence of a successful transfer by simply posting operational software in the internet-based Apple/Google stores.  That option did not exist for the prior cohort where Grantees had to secure partnerships with external software providers to achieve transfer outcome.

Less than one quarter of the projects engaged an external corporate partner at implementation.  This requires convincing a partner that the envisioned output has sufficient value to expend limited corporate resources on deployment and support, thereby staking the company’s future viability on some potential return. Of those that did engage companies, their eventual success depended heavily on the mutual relationship.  Grantees with long-term relationships had the highest demonstrated success.  One Grantee achieved a high degree of success by designating the president of the partner company as the lead investigator for all three of their development projects.  Another established relationship allowed the Grantee to rapidly transfer software components directly into upgrades to an existing commercial product as new features/functions.  Some investigators formalized intellectual property and licensing agreements with their business partner at the project’s inception, allowing all participants to focus on the goal.  Strong corporate collaborations free up academics to focus on their scholarship.

Some investigators greatly overestimate their ability to achieve their promised outcomes.  One Grantee implemented three interdependent projects to create a comprehensive inventory tracking and retrieval system to facilitate employment in commercial warehouses.  The actual output was essentially a spreadsheet.  In multiple cases, the lack of progress after two or three years should have been cause to reallocate the funding to more productive activities.  Few investigators integrated formal decision gates within their implementation plans, so they lacked mechanisms to seriously scrutinize their progress.  Some erroneously believe that the initial peer review and award was all the validation they needed for the entire project’s timeframe.

DISCUSSION

Overall, Grantees have improved their record of delivering on intended results, but too many still fall far short of the intended outcomes that were deemed worthy of funding by peer reviewers.   What factors might account for the persistent disparity in the results achieved by development projects intending transfer outcomes?

The Proposal Process:  

In an effort to impress review panels, Grantees often exaggerate the likely progress, results and value of development projects.  Proposals often lack supporting evidence to validate the need, that the proposed solution is novel and feasible, or that they have planned the path to transfer.  One SBIR Phase II did not explore the target customers interest in the proposed commercial software product until after it was an operational prototype, and then only to find out that they rejected the entire concept of the product outright.

At the same time, Grantees often allocate insufficient personnel time and resources to accomplish the intended results.   It is difficult to envision how project continuity can be maintained when key people are dedicating two to four hours per week (5% - 10% FTE).  This level of effort does not compare favorably to corporate approaches to new product development where teams of people dedicate the majority of their time within a framework of defined milestones and deliverables, and under the supervision of multiple layers of managers.  Grantees even assign part-time graduate students to critical projects tasks in the misguided effort to build capacity, despite their obvious lack of experience.  Numerous investigators view technology transfer as an unfunded mandate.  They cite the absence of proposal points reserved for preliminary market research, corporate partnerships, or preliminary technology transfer plans.  Expecting Grantees to generate technology transfer plans during their first year of funding is tantamount to saying, ‘Ready, Fire, Aim’ – because staff and resources are already being expended.

Grant Implementation:

 Some Grantees do recognize the importance of applying business and industry methods to development projects, and have either made the effort to secure that expertise or are open to guidance on relevant topics.  However, too many Grantees do not demonstrate diligent implementation once funding is secured.  Often the narrative regarding progress provided in their first year’s APRs contain a generic description of the project taken from the original proposal.  Others may insert some original narrative in the first year’s APR then paste that same narrative in one or more subsequent APRs.  Despite their obvious absence of progress, these same APR’s report the project as ‘On Time’ and ‘No Problems.’   Grantees are keenly aware that reporting delays or problems invites scrutiny from Project Officers.  This casual approach to reporting and monitoring does not reflect best practices in product development, where project managers closely monitor progress, and cancel projects or even terminate staff when planned progress is not demonstrated.

Some investigators candidly describe similar behaviors in the personal interviews.  Once funding is secured, daily work demands and career incentives take precedent over grant implementation.  Department chairs prioritize course load, student advisement and committee assignments over grant activity.  The time devoted to a grant is weighted towards research activity over development activity because the former will generate data suitable for a scholarly manuscript necessary for tenure or to secure additional funding.  The investigator’s local status is also an issue.  Junior faculty and graduate students are unlikely to demand senior faculty member pay attention to a stalled project.  The study team also learned that some lead investigators still harbor the antiquated notion that collaborating with companies can somehow jeopardize their academic integrity, but again project staff are unwilling to correct a perception even though it creates formidable barriers to successful technology transfer.

Many investigators – especially academic faculty – still don’t know what they need to know about invention, transfer and innovation, and are reluctant to devote time to learn.  Some avoid recruiting external business expertise due to ego, fear of losing control or sharing the available funds.  Some investigators are still not even involving their own Technology Transfer Offices (TTO), even though TTOs have crucial expertise and are usually free to access. While Principal Investigators participate in annual meetings where NIDILRR directs them to available resources and briefs them on relevant concepts, this information is not routinely passed along to the co-investigators, staff and graduate students who end up being chiefly responsible for project implementation. 

The Grant Review Process  

The study team’s review of Grantee proposals and/or interviews with the investigators, reveals numerous instances where the proposal review panel as well as NIDILRR’s internal proposal review, could have raised more questions about the applicant’s claims. For example, an SBIR Phase II application that projected skyrocketing cash flow from a new application without evidence from related products is an obvious candidate for closer scrutiny.  And in fact, there is no further mention of cash flow in its subsequent progress reports.  The review process should verify that the stated problem exists and that the proposed solution is novel.  While research projects require preliminary data to validate this issue, development projects often make unchallenged claims about novelty.  In several cases, a quick Internet search by the study team revealed existing products nearly identical to the proposed and funded outputs, all of which failed to achieve their outcomes.

Grant Management Process:

The sponsor bears ultimate responsibility for the outcomes achieved by the Grantees.  NIDILRR’s regulations expressly state that funding for future years is dependent on demonstrating substantial progress during the current year.  NIDILRR has the authority to delay funds or even withhold them entirely, yet that action has almost never been taken, and certainly did not occur with any of the Grantees in the current study.  Along with the examples of continued progress and eventual success, there are multiple examples where little or no progress was made over multiple years, yet the funds continued to flow.  NIDILRR’s Project Officers are in a role equivalent to upper management in private corporations, yet the evidence shows that they do not all provide the oversight necessary to ensure progress is made and outcomes achieved.  The study team found multiple cases where even a cursory review of Grantee progress could have led to closer scrutiny/guidance or even the curtailment of funds.  However, there were no apparent examples where the Project Officer exerted corrective actions for languishing projects.  While some Grantees reported regular communication with their Project Officers, others reported no personal contact once the funds were awarded. Apparently, some NIDILRR staff view grants and cooperative agreements as unrestricted gifts despite the explicit requirements for progress.

Public Role Implications  

NIDILRR’s mission is to fund research and development activity that generates beneficial impacts on the quality of life for Persons with Disabilities.  Development projects offer the most visible and shortest-term opportunities to generate such beneficial impacts.  It is up to persons with disabilities and their related stakeholders to document the intended outcomes of funded proposals and to hold the NIDILRR community accountable for the efficient and effective use of public monies expended to accomplish that mission.