FEASIBILITY STUDY OF USABILITY ASSESSMENT ON QUALITY OF LIFE TECHNOLOGY USING GUIDELINES IN ISO 9421-11 AND COMMON INDUSTRY FORMAT (CIF)
Shinyoung Lim, Jongbae Kim, Jeong-Hyun Kim, Se-Eun Yang
Korea National Rehabilitation Research Institute
ABSTRACT
Usability assessment has been installed into a wide range of software that focuses on assessing product usage from the user's perspective. Usability assessment of the quality of life technology for individuals with disability is being discussed and tentatively designed relevant to apply to products of quality of life technology. Designing an appropriate usability assessment method by referencing the currently available International standards on software usability tests has number of considerations and adjustments to produce valuable feedbacks for effective product quality enhancement. The feasibility study on designing a usability assessment method into quality of life technologies is presented to invite calls for further discussion.
KEY WORDS
Quality of life technology, usability assessment, and matching user needs and functional implementations
INRODUCTION
The definition of usability is extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [1]. Here, the meaning of effectiveness stands for accuracy and completeness with which users achieve specified goals and the meaning of efficiency is resources expended in relation to the accuracy and completeness with which users achieve goals. The meaning of ‘context of use’ is for users, tasks, equipment including hardware, software and materials, and physical and social environments in which a product is used. However, the meaning of satisfaction includes, but not limited, satisfaction scale, questionnaire, and discretionary usage.
At present, the usability assessment is adopted in the overall process of software development from its life cycle, development, product, and product being used in real world [2, 3, 4, 5, 6]. But most of the usability assessment is being implemented for non-disabled individuals who are going to use the software products. When it comes to applying the usability assessment to the process of developing products for the individuals with disability, number of considerations and adjustments is required to fit into the environment for the prospective users. Authors collect related international standards on usability assessment to review the specifications of the usability protocols and restructure protocol specifications to adopt the usability assessment process for the individuals with disability.
The considerations and adjustments as well as design of the usability assessment protocol suitable for the products for the individuals with disability are discussed in this paper with further research plans.
ISSUES ON USABILITY ASSESSMENT
As a quality of life technology project in South Korea matures in its second year, it will be required to shift research teams’ focus from research and development to usability assessment of the prototypes from the project. The ultimate goals of the usability assessment are to deliver assessment feedback to research teams on inconsistency in user needs from the development outcomes, and establishing an evidence base that will provide research teams with a relevant analysis method of cost-benefit estimation as well as user acceptance and satisfaction.
The research and development teams of the quality of life technology project in South Korea realize that there is a large and growing gap between the prototype features and user acceptance. Three influencing factors have been identified as a disconnect between the gap: 1) inconsistency in deploying concept of accessibility into development process for the prospective users, i.e., individuals with specified disability, 2) mismatches between user needs and prototype features, and 3) lack of evidence data that will be used as feedbacks to the prototype improvement to reduce the gap. Research team members regard usability assessment as one of available methods of resolving these three influencing factors.
ISO USABILITY ASSESSMENT
ISO Usability Framework
Figure 1. Usability Framework dIn order to specify or measure usability it is necessary to identify the goals and to decompose effectiveness, efficiency and satisfaction and the components of the context of use into sub-components with measurable and verifiable attributes. The components and the relationships between them are illustrated in figure 1 [7, 8, 9, 10, 11, 12].
When specifying or measuring usability, the following information is needed:
- a description of the intended goals;
- a description of the components of the context of use including users, tasks, equipment, and environments. This may be a description of an existing context, or a specification of intended contexts. The relevant aspects of the context and the level of detail required will depend on the scope of the issues being addressed. The description of the context needs to be sufficiently detailed so that those aspects of the context which may have a significant influence on usability could be reproduced;
- target or actual values of effectiveness, efficiency, and satisfaction for the intended contexts.
Description of goals
The goals of use of a product should be described. Goals may be decomposed into subgoals which specify components of an overall goal and the criteria which would satisfy that goal. For example, a telephone sales clerk might have the goal to “Maintain customer orders”. This overall goal might then be decomposed into subgoals such as:
- “Make accurate record of all orders placed by customers”;
- “Provide information rapidly in response to customer inquiries about orders placed”.
The level at which the overall goal is set is a function of the boundary of the work system which is under consideration and which provides the context of use. In the example above, the work system under consideration consists of clerks taking telephone orders.
Context of use
Relevant characteristics of the users need to be described. These can include knowledge, skill, experience, education, training, physical attributes, and motor and sensory capabilities. It may be necessary to define the characteristics of different types of user, for example users having different levels of experience or performing different roles.
Choice of measures
It is normally necessary to provide at least one measure for each of effectiveness, efficiency and satisfaction.
Because the relative importance of components of usability depends on the context of use and the purposes for which usability is being described, there is no general rule for how measures should be chosen or combined.
The choice of measures and the level of detail of each measure are dependent on the objectives of the parties involved in the measurement. The relative importance of each measure to the goals should be considered. For example where usage is infrequent, high importance may be given to measures of learning and re-learning.
If it is not possible to obtain objective measures of effectiveness and efficiency, subjective measures based on the user's perception can provide an indication of effectiveness and efficiency.
Satisfaction measures the extent to which users are free from discomfort, and their attitudes towards the use of the product. Satisfaction can be specified and measured by subjective rating on scales such as discomfort experienced, liking for the product, satisfaction with product use, or acceptability of the workload when carrying out different tasks, or the extent to which particular usability objectives (such as efficiency or learnability) have been met. Other measures of satisfaction might include the number of positive and negative comments recorded during use. Additional information can be obtained from longer-term measures such as rate of absenteeism, observation of overloading or underloading of the user's cognitive or physical workload, or from health problem reports, or the frequency with which users request transfer to another job.
DESIGN OF USABILITY ASSESSMENT FOR QUALITY OF LIFE TECHNOLOGY
Considerations of Usability Assessment Update
With the ISO usability assessment protocol suites, authors search the appropriate approach to design a new usability assessment for the overall process, especially in the prototype process, of quality of life technology. Number of considerations to restructure the usability assessment protocol suites is investigated as follows:
- Current features of the ISO usability assessment protocols are required to adopt the prospective user’s capabilities due to their disability and residual functions.
- A certain specified process that reflects the limitations of the prospective user’s capabilities in the development process along with the ISO usability assessment protocols.
- Addition of a defined process of hearing and collecting feedbacks and comments on overall research and development process from the individuals with disability, i.e., the prospective users is required.
- Addition of a verifying process of quality of product, which includes matching the user needs with the outcomes of research prototypes.
- Additional features of risk management including prevention of secondary complications and getting injuries due to the product usage are required as a safety requirement.
- As the usability assessment is new to the rehabilitation and quality of life technology communities, a reference model of the usability assessment will be helpful.
-
As the ultimate goal of the usability assessment is to apply the assessment protocol suites to all the process of product life cycle as shown in the figure 2 with a few ISO standards, the newly designed usability assessment protocol suites will be applied to each process of the life cycle processes to effect the product’s quality of usability [1, 2].
Figure 2. Ultimate goal of usability assessment (ISO perspective) d
Design of the Usability Assessment for Quality of Life Technology
At present, authors tentatively design a reference model of usability assessment consists of number of unit protocols. The protocol suites of the usability assessment for quality of life technology are for product in use status, not for overall status in figure 2.
Figure 3. Reference Model of Usability Assessment for Quality of Life Technology dThe framework of the newly designed usability assessment reference model is shown in figure 3 which consists of three major steps: preliminary step, assessment step, and interpretation step.
Preliminary Step of the Usability Framework
This step is for preparing the usability assessment of a certain quality of life technology product, which consists of an IRB approval including interactions with participants, i.e., subjects, regarding explanation of reason and method of the usability assessment followed by acquiring signature on the consent form. This step also includes disability assessment of the subject and survey on similar experience, which will affect the usability assessment. For instance, in case of usability assessment on mobile computing, we need to ask the subject if he or she has experience of using similar devices for a certain amount of time. We also need to ask the subject of joining similar usability assessment sessions prior to this participation.
Assessment Step of the Usability Framework
This step consists of number of surveys, on-site instruction of product usage, and product usage sessions. The prologue survey consists of preliminary questionnaires on the product being assessed, and the epilogue survey consists of the questionnaires focused on the particular features of the product being assessed as well as features that generates convenient, attraction, comfort and discomfort to the subject. We also recommend time period of product usage session be more than two weeks allowing the subject get used to the product being assessed.
Interpretation Step of the Usability Framework
This step is for reporting the usability assessment of a certain quality of life technology product, which consists of an in depth interview, observation and media recording of a number of tasks by the subject, a final survey, and analysis and evaluation sessions. The in depth interview is for collecting subject’s comments and opinions those are not asked as open questions. The observation and media recording session is for acquiring evidence data on product of quality in use from the subject side that consist of effectiveness, productivity, safety, and satisfaction of the product. The final survey is for assessing accessibility, convenience, portability, error frequency, and satisfaction factors of the product being assessed. The analysis and evaluation session is the final unit protocol of the reference model of the usability assessment for the products from the quality of life technology. We recommend Common Industry Format (CIF) based usability assessment report which is the outcome of the last protocol unit [13, 14, 15, 16].
CONCLUSION
Authors introduce a novel method of usability assessment for the products from the quality of life technology by adopting the ISO International standard of usability assessment. A number of considerations and adjustments are also discussed to develop a newly designed usability assessment in tentative status. The reference model of the usability assessment is presented to invite further discussion on its value and real world implementation.
DISCUSSIONS
For the further study on the newly designed usability assessment, we need to review the reference model of the usability assessment and assess the model based on time and cost effectiveness in terms of improving quality of product as well as quality of life for the individuals with disability.
REFERENCES
[1] International Standard ISO 9241-11 (1998) Ergonomic requirements for office work with visual display terminals (VDTs) Part II: Guidance on usability: ISO.
[2] Nigel Bevan and Ian Curson (1999). Planning and Implementing User-Centered Design. Proceedings of 1999 Conference on Human Factors in Computing Systems (CHI’99) (pp. 137-138). Pittsburgh, PA. 15-20 May 1999: ACM.
[3] International Standard ISO 9241-151 (2008). Ergonomic of human-system interaction – Part 151: Guidance on World Wide Web User Interfaces: ISO.
[4] Sarah A. Douglas, Arthur E. Kirkpatrick, I. Scott MacKenzie (1999). Testing Pointing Device Performance and User Assessment with the ISO 9241, part 9 Standard. Proceedings on CHI’99 (pp. 215-222). 15-20 May 1999: ACM.
[5] Kate S. Hone and Robert Graham (2001). Subjective Assessment of Speech-System Interface Usability. Proceedings on Eurospeech 2001 – Scandinavia (pp. 2083- 2086). 3-7 sep. 2001. Aalborg, Denmark: ISCA.
[6] I. Padayachee, P. Kotze, A. van Der Merwe (2010). ISO 9126 external systems quality characteristics, sub-characteristics and domain specific criteria for evaluating e-Learning systems. Proceedings of South African Computer Lecturers’ Association (SACLA) 2010. 3-5 June 2010. Pretoria, South Africa: ACM.
[7] Technical Report ISO/IEC TR9126-4 (2004) Software Engineering – Product Quality Part 4: Quality in user metrics: ISO.
[8] Nigel Bevan (2009). Extending Quality in Use to Provide a Framework for Usability Measurement. Proceedings of Human Computer Interaction (HCI) International 2009 (pp. 13-22). 19-24 July 2009. San Diego, California: Springer V erlag.
[9] Martin Maguire (2001). Methods to support human-centered design. Int. J. Human-Computer Studies, 55, 587-634.
[10] International Standards ISO/IEC 9126-1 (2001). Software engineering – Product quality – Part 1. Quality Model: ISO.
[11] Jonathan Earthy, Brian Sherwood Jones, Nigel Bevan (2001). The improvement of human-centered processes – facing the challenge and reaping the benefit of ISO 13407. International Journal of Human-Computer Studies, 55, 4, 553-585.
[12] Alain Abran, Adel Khelifi, Witold Suryn, Ahmed Seffah (2003). Usability Meanings and Implementations in ISO Standards. Software Quality Journal, 11, 323-336.
[13] NIST, NISTIR 7742 (2010), Customized Common Industry Format Template for Electronic Health Record Usability Testing: NIST.
[14] NIST (2001). Common Industry Format for Usability Test Reports: NIST.
[15] Nigel Bevan (1998). Common Industry Format Usability Tests. Proceedings on UPA’98, Usability Professionals Association. Scottsdale, Arizona. 29 June-2 July, 1999: UPA Press.
[16] Nigel Bevan, Nigel Claridge, Martin Maguire, Maria Athousaki (2002). Specifying and evaluating usability requirements using Common Industry Format. Proceedings of IFIP 17th World Computer Congress (pp. 133-148). Montreal, Canada, 25-30 August 2002: Kluwer Academic Publishers.
ACKNOWLEDGEMENTS
This work was supported by the Technology Innovation Program (100036459, Development of center to support QoLT industry and infrastructures) funded by the MKE/KEIT, Korea.