... CONTENT-ORIENTED VALIDATION CONTENT-ORIENTED VALIDATION OF THE FUNCTIONAL COGNITIVE ASSESSMENT Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Alyssa A. Ford, MOT, OTR Copyright September 5, 2017 By: Alyssa A. Ford, MOT, OTR All rights reserved Approved by: Candace Beitman, EdD, OTR Committee Chair Beth Ann Walker, PhD, OTR Committee Member Kim Warchol, BS, OTR/L Committee Member Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis CONTENT-ORIENTED VALIDATION ii Abstract The Functional Cognitive Assessment is a standardized cognitive performance assessment that is criterion referenced and administered during everyday tasks (Ebell, Ford, & Warchol, 2016). Psychometric testing is needed in order to establish content-oriented validity evidence and utility of the Functional Cognitive Assessment; thus, the purpose of the study was to establish the content-oriented validity evidence of the Functional Cognitive Assessment. Ten subject matter experts anonymously responded to a survey comparing the test items to the construct of functional cognition. Interrater agreement was 0.90 for representativeness and 0.70 for clarity. Item level content validity indices ranged from 0.70-0.90 for representativeness. The scale level content validity index was 0.81 for representativeness. Factor validity index ranged from 0.90-1.0 for each subtask. The overall factor validity index was 0.98. Item content validity indices for representativeness were assessed using a multi-rater kappa statistic, which ranged from 0.66-0.90 for each subtask, indicating that the subtasks ranged from excellent to good. Results support initial content-oriented validation of the Functional Cognitive Assessment. Keywords: cognitive disabilities model, Alzheimers disease and related dementias, performance assessment, content-oriented validity evidence, content validity, functional cognition CONTENT-ORIENTED VALIDATION iii Acknowledgements I would like to acknowledge the guidance, support, and encouragement that my parents, John Hawkins and Marianne Hawkins provided throughout my entire academic career. They ensured that I had access to the resources and education that I needed throughout every stage of my education. They instilled a value of work-ethic, love for learning, and lifted me up when I was down. Thank you to my siblings for encouraging me every step of the way. Jonathan Hawkins, is my closest sibling in age and always set the bar high. Leah Scott was my sounding board during the past year while we simultaneously engaged in coursework for separate academic endeavors. I thank Mary Comparato for reviewing my application for the Early Investigator Scholarship for Dementia Research and Study. I thank Stephen Hawkins for his support during the data analysis stage of this doctoral project. I would also like to acknowledge my mother-in-law, Joyce Ford for her support throughout the Doctor of Health Science Program and during completion of this doctoral project. My sons, Anderson and Aaron Ford, inspire me to be an ethical and efficient role model and to always complete the best work that I am possible of accomplishing. Thank you to my husband, Andrew Ford, for being by my side through every stage of the Doctor of Health Science program and this doctoral project. I would not have been able to complete the course work or this doctoral project without his instrumental support on a daily basis. Andrew Ford is a loving and engaged husband and father who patiently performed more than his fair share of the family responsibilities while I pursued my academic goals. For this, I will forever be grateful. I would also like to acknowledge several colleagues who have been pivotal in my career. Thank you to Kim Catlett, Lora Alonzo, and Ryan Palombit for modeling true interprofessionalism in the care of clients who have Alzheimers disease or related dementias. CONTENT-ORIENTED VALIDATION iv Not only are they colleagues, they were role models during my early years as an occupational therapist. It was with this team that I began to understand how to incorporate the cognitive disabilities model into occupational therapy practice. I must also acknowledge Kathryn McCormick for her support during the early stage of this doctoral project. A fellow occupational therapist that Id like to recognize is Sarah King, who has been a close colleague as I have solidified my understanding of the cognitive disabilities model and completed my coursework and doctoral project. Sarah was always willing to hear about my educational pursuits. Thank you to C. Ann Irwin for her support, which has enabled me to have access to exceptional continuing education and opportunities to disseminate my knowledge. Finally, thank you to the pilot reviewers and all of the respondents who completed the survey for this doctoral project. Id like to express my deep gratitude to my doctoral project committee at the University of Indianapolis. Dr. Candace Beitman was my graduate school advisor and later agreed to be the committee chair for my doctoral project. Dr. Beitman provided guidance during the doctoral project proposal process, and patiently provided support throughout the doctoral project. Her kind words of encouragement seemed to come at the right moments. Thank you to Dr. Beth Ann Walker for being the analysis expert on my committee. I was grateful for her critical eye and analytical skills. She went above and beyond as an analysis expert for my doctoral project and made herself available whenever I needed to discuss small and large details. Thank you to Kim Warchol for providing crucial continuing education to healthcare professionals who treat clients who have Alzheimers disease or related dementias. Her continuing education courses formalized my understanding of the cognitive disabilities model and its application to occupational therapy. I am grateful that our professional paths crossed and that we have had opportunities to collaborate on projects. I will always be grateful that Kim involved me in the CONTENT-ORIENTED VALIDATION creation of the Functional Cognitive Assessment and agreed to be the content expert for my Doctoral Project. Finally, thank you to the Alzheimers Association Greater Indiana Chapter for awarding me the 2015 Early Investigator Scholarship for Dementia Research and Study. People with Alzheimers disease and related dementias deserve to live the highest possible quality of life. Thank you for the financial support and believing in the merits of this doctoral project. v CONTENT-ORIENTED VALIDATION vi Table of Contents Abstract ......................................................................................................................................... ii Acknowledgements ....................................................................................................................... iii List of Tablesviii INTRODUCTION ........................................................................................................................ 1 LITERATURE REVIEW ............................................................................................................. 5 Cognitive Disabilities Model ........................................................................................................ 5 Allen Cognitive Levels ................................................................................................................. 6 International Classification of Functioning, Disability and Health Model ................................... 14 Similarities between the CDM and ICF-M ................................................................................... 17 Theory of Retrogenesis ................................................................................................................. 18 The Allen Battery of Assessments ................................................................................................ 18 The Functional Cognitive Assessment.......................................................................................... 21 Content-Oriented Validity Evidence............................................................................................. 23 METHODS ................................................................................................................................... 24 Research Design............................................................................................................................ 24 Sampling ....................................................................................................................................... 24 Instrumentation ............................................................................................................................. 26 Pilot Study..................................................................................................................................... 27 The Survey .................................................................................................................................... 27 Plan for Data Analysis .................................................................................................................. 29 RESULTS ..................................................................................................................................... 32 Interrater Agreement ..................................................................................................................... 32 CONTENT-ORIENTED VALIDATION vii Content Validity Index (representativeness) ................................................................................ 33 Factor Validity Index .................................................................................................................... 33 I-CVI using a Modified Kappa Statistic ....................................................................................... 34 Respondent Feedback ................................................................................................................... 35 DISCUSSION ............................................................................................................................... 36 Potential Use of the FCA .............................................................................................................. 38 Limitations .................................................................................................................................... 39 Conclusion .................................................................................................................................... 40 REFERENCES ............................................................................................................................. 41 Appendix A ................................................................................................................................... 55 Appendix B ................................................................................................................................... 56 Appendix C ................................................................................................................................... 57 CONTENT-ORIENTED VALIDATION viii List of Tables Table 1...34 CONTENT-ORIENTED VALIDATION 1 Content-Oriented Validity Evidence of the Functional Cognitive Assessment It has been estimated that 5.2 million Americans aged 65 years or older have Alzheimers disease (AD) (Herbert et al., 2013). The incidence of AD amongst this population is projected to nearly triple by 2050, which could yield up to 13.8 million cases unless a cure or prevention is found (Herbert et al., 2013). As cited in the 2012 Alzheimers Disease Facts and Figures (Alzheimers Association, 2012), adults with Alzheimers disease or related dementias (ADRD) have more than triple the amount of hospital stays as compared to other older adults (Bynum, 2011). Risks encountered by people with ADRD who live alone or in the community may include: malnourishment (Nourhashemi, Amouyal-Barkate, Gillette-Guyonnett, Cantet, & Vellas, 2005); nursing home placement (Yaffe et al., 2002); unmet social, environmental, psychological, and medical needs (Miranda-Castillo, Woods, & Orrell, 2010); disorientation or self-neglect resulting in harmful incidents or emergencies (Tierney et al., 2007; Tierney et al., 2004); falls (Rubenstein & Josephson, 2006); injuries and wandering away from home while unattended (Rowe et al., 2010), fatal injuries (Kibayashi, Sumida, Shojo, & Hanada, 2007); and psychiatric symptoms such as depression, agitation, and psychosis (Apostolova & Cummings, 2008; Lehmann, Black, Shore, Kasper, & Rabins, 2010). In a population-based sample, Steinberg et al. (2003) noted mental and behavioral symptoms such as delusions, apathy, and aberrant motor behavior. Lehmann et al. (2010) noted that a lack of awareness about cognitive dysfunction, functional deficits, and psychiatric symptoms increase the risk of adverse outcomes in people who have dementia and live alone. It is clear that inadequate support of the person with ADRD can lead to unfortunate and disastrous consequences. The risks associated with CONTENT-ORIENTED VALIDATION 2 inadequate support are further complicated when people who have ADRD and their caregivers are unaware of the condition (Alzheimers Association, 2015). A lack of understanding about ones medical condition and the risks of inadequate supervision during day to day tasks may lead to expensive, preventable hospitalizations. Preventable hospitalizations are those which could have been avoided with better access or higher quality of preventive or primary care. The Office of Disease Prevention and Health Promotion set a goal of reducing preventable hospital admissions for people with ADRD by 10% by 2020 (Office of Disease Prevention and Health Promotion, 2014). Community-dwelling individuals with dementia are more likely than those individuals without dementia to have a potentially preventable hospitalization or an emergency department visit that resulted in a hospitalization (Feng, Coots, Kaganova, & Wiener, 2014). In addition, a substantial number of hospitalizations and emergency department visits prior to and during the last year of life were shown to be potentially avoidable (Feng et al., 2014). Individuals with dementia, as well as comorbid dementia and depression are a particularly at risk population of individuals who may benefit from interventions to reduce preventable hospitalizations (Davydow et al., 2014). Occupational therapists and other professionals are positioned to provide support to people with ADRD and their caregivers. In fact, occupational therapy (OT) has been found to be an essential and effective element in discharge planning (Renda, Lee, Keglovits, & Somerville, 2016). The Alzheimers Disease 2016 Facts and Figures (Alzheimers Association, 2016) lists a variety of interventions that can be offered to caregivers, such as case management, psychoeducational, counseling, support groups, respite, psychotherapeutic approaches, and CONTENT-ORIENTED VALIDATION 3 multicomponent approaches (Pinquart & Srensen, 2003; Srenson, Duberstein, Gill, & Pinquart, 2006). However, none of these approaches explicitly describes the provision of oneon-one intervention with the client who has ADRD during activities of daily living, instrumental activities of daily living, leisure tasks, etc., nor education to the caregiver about the clients task performance and how to provide verbal, visual, and tactile cues to the client in order to facilitate the best possible performance during these tasks. Based on the Occupational Therapy Practice Framework: Domain and Process, 3 rd edition (OTPF-III), (American Occupational Therapy Association, 2014) client factors are capacities, characteristics, or beliefs that influence performance in occupations. Body functions is a category of client factors, and cognition falls within the mental functions (American Occupational Therapy Association, 2014). OT practitioners are particularly concerned with how cognition affects performance skills in occupations such as activities of daily living (ADLs), instrumental activities of daily living (IADLs), rest and sleep, education, work, play, leisure, and social participation. OT practitioners using the Cognitive Disabilities Model (CDM) (Allen et al., 1995; Allen et al., 1992) assess functional cognition in the context of the above stated areas. Functional cognition is how an individual utilizes and integrates his or her thinking and processing skills to accomplish everyday activities in clinical and community living environments (American Occupational Therapy Association, 2016, para. 3). Additionally, occupational therapists have described that functional cognition . . . encompasses both functional performance and the global cognitive processing capacities of the brain. Functional performance arises from the interaction between global cognitive processing capacities reflected in what a person pays attention to and the activity demands of specific functional tasks, e.g. the motor and verbal skills, social behaviors, self-awareness, and awareness of contexts required for CONTENT-ORIENTED VALIDATION 4 performing various tasks (Allen, Austin, David, Earhart, McCraith, & Riska-Williams, 2007, p. 7-8). An Allen Cognitive Level (ACL) is a descriptor of a clients cognitive abilities (Allen, 1982, Allen et al., 1995; Allen et al., 1992). OT practitioners using the cognitive disabilities model may initially administer the Allen Cognitive Level Screen-5 (ACLS-5) or Large Allen Cognitive Level Screen-5 (LACLS-5) (Allen et al., 2007). The authors clearly describe that the ACLS-5 and LACLS-5 were designed to be a quick estimate of undetected problems related to functional cognition or for providing an estimate of the severity of the functional cognition deficits. The score should be used by the OT practitioner or other skilled clinician in order to effectively select additional assessments and in the overall interpretation of the clients cognitive status (Allen et al., 2007). Therapists who use the CDM also use non-standardized skilled observation to identify and describe specific patterns of behavior associated with each cognitive level during any functional task (Allen et al., 1992; McCraith, et al., 2011). Ebell, Ford, and Warchol (2016) developed a tool titled The Functional Cognitive Assessment (FCA) to guide therapists skilled observations and the determination of a clients ACL through functional task analysis. The FCA is a standardized, criterion-referenced test in which the client performs a variety of functional tasks (Ebell et al., 2016). The FCA consists of 10 functional tasks, including three ADLs, four IADLs, and three leisure activities. Test administrators use scoring rubrics to identify observable behaviors during completion of the functional tasks that are consistent with ACLs. See Appendix A for a copy of the FCA at the time of this study. CONTENT-ORIENTED VALIDATION 5 In order to achieve the American Occupational Therapy Associations 2017 Centennial Vision, Doucet, Woodson, & Watford (2014) recommended that OT practitioners focus on quantifying and centering on occupational-based practice. In a systematic review of studies on measurement properties of evaluation instruments for adults, Yuen and Austin (2014) stated that methodologically strong content validity articles explored content validity by assessing relevance and representativeness of potential test items. Furthermore, they discussed that studies related to assessment development could be strengthened by including theoretical underpinnings of the assessments construct. Yuen and Austin (2014) concluded by stating that the implementation of psychometric research will support the goal to be recognized as a scientific and evidence-based profession. The next step is to determine if the FCA demonstrates evidence of content-oriented validity. Therefore, the purpose of this study was to establish content-oriented validity of the FCA. Literature Review Cognitive Disabilities Model The American Occupational Therapy Association (AOTA) (2013) identified the CDM as a model that addresses cognition and occupational performance in an evidence-based manner in Cognition, cognitive rehabilitation, and occupational performance. Allen et al. (1992, p. 102) stated that by understanding a clients cognitive abilities, a therapist can adapt activities so that the clients cognitive abilities are continuously maximized. According to the OTPF-III (2014), activity analysis is important because the process helps OT practitioners understand the demands of an activity. Therefore, the aforementioned concept of establishing task equivalence and providing caregiver training about task equivalence is consistent with the OTPF-IIIs (2014) CONTENT-ORIENTED VALIDATION 6 description of OT interventions such as education and training, as well as approaches to intervention, such as maintenance, modification (compensation and adaptation), and prevention. As cited in the OTPF-III (2014), the intervention of modification/compensation/adaptation has been described as finding ways to revise the current context or activity demands to support performance in the natural setting, [including] compensatory techniques, [such as] . . . enhancing some features to provide cues or reducing other features to reduce distractibility (Dunn, McClain, Brown, & Youngstrom, 1998, p. 533). By understanding the clients difficulties with learning and problem solving, therapists may be able to anticipate hazards in the environment and prevent unsafe or undesirable situations (Allen et al., 1995; Allen et al., 1992; Allen, 1999). Allen et al. (1992) postulated that the therapist is responsible for discharge recommendations, which may include training caregivers to provide opportunities for the client to use their cognitive abilities and warning the client and caregivers about potentially harmful situations if safeguards are not put into place. Allen Cognitive Levels Allen (1982) initially developed six cognitive levels which represent a hierarchy of abilities. The determination of a clients Allen Cognitive Level (ACL) is based on clinically observable and qualitative differences in his or her abilities during screening, testing, and functional activities (Allen et al., 1995; Allen et al., 1992; Allen, 1999; McCraith et al., 2011). Allen Cognitive Levels 1-5 are further defined into performance modes (Allen et al., 1995; Allen, et al., 1992; Allen, 1999; McCraith et al., 2011). Because Level Six is the highest level, representing normal cognition, there was no need to refine it into performance modes CONTENT-ORIENTED VALIDATION 7 (Allen et al., 1992). The performance modes were developed to be more specific measures of cognitive abilities and are formally described as .0, .2, .4, .6, and .8 within the Allen Cognitive Levels (Allen et al., 1992). There are functional descriptors associated with the ACLs and performance modes. Clients who function in .8 modes begin to demonstrate abilities in the next level. However, these higher abilities at .8 are inconsistent and clients can easily become frustrated if they are expected to perform at the next higher performance mode. Clients who function in .0 modes demonstrate a shift from the previous level. (Allen et al, 1992; McCraith et al., 2011). When therapists have an understanding of a clients ACL, they can develop appropriate goals, treatment plans, and discharge recommendations (Allen et al., 1995; Allen et al., 1992; McCraith et. al., 2011). Allen Cognitive Level One comprises the lowest level of cognitive abilities, while Allen Cognitive Level Six (ACL 6) describes normal cognitive function. Allen Cognitive Level One is titled Automatic Actions (Allen et al., 1992; McCraith et. al., 2011). Clients functioning in this level present with reflexive abilities and can be positioned to sit with support (Allen et al., 1995; Allen et al., 1992; McCraith, et al., 2011). They require total cognitive assistance and 24-hour supervision via nursing care for all aspects of self-care (Allen et al., 1995; Allen et al., 1992). Communication may range from moans and grimaces to smiles and increased responses to loved ones. The client may initiate communication in response to pain or external stimuli (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). Allen Cognitive Level Two is titled Postural Actions (Allen et al, 1995; Allen et al., 1992; McCraith et al., 2011). Clients functioning in this level may present with gross motor abilities, such as the ability to sit, stand, and walk (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). Within this level, there is also an emerging use of the hands, such as the CONTENT-ORIENTED VALIDATION 8 ability to grab onto a bar to prevent falling (McCraith et al., 2011). Clients may wander aimlessly; thus, caregivers may engage in environmental modifications to prevent falls and unsafe wandering (McCraith et al., 2011). Individuals may require maximum cognitive assistance for 24 hour supervision via nursing care to prevent falls during gross motor activities (Allen et al., 1995; Allen et al., 1992). Clients in this level may recognize and state their names, use perseverative words, and use short phrases and gestures (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). Allen Cognitive Level Three is titled Manual Actions (Allen et al, 1995; Allen et al., 1992; McCraith et al., 2011). People functioning in Allen Cognitive Level Three have the use of their hands, which may be demonstrated in the ability to grasp, manipulate, and attempt to use objects for their intended purpose. Clients may also exhibit an understanding of task completion upon the utilization of all available objects, and may be able to communicate their needs and name familiar objects and actions (Allen et al., 1995; Allen et al., 1992; McCraith et. al., 2011). Individuals may not differentiate between day, date, or time; however, they may be able to acknowledge the difference between their home and the hospital. People functioning in Allen Cognitive Level Three require moderate cognitive assistance and 24 hour supervision for cues through the steps of an activity to avoid potentially dangerous situations (Allen et al., 1995; Allen et al., 1992). Allen Cognitive Level Four is titled Goal-Directed Actions (McCraith et al., 2011). Individuals functioning in Allen Cognitive Level Four have the ability to sequence themselves through the steps of a simple, routine task. Based on their performance mode, task quality may be degraded. However, they may also complete a goal with good quality. There is a continuum of problem-solving abilities within this level (Allen et al., 1995; Allen et al., 1992; CONTENT-ORIENTED VALIDATION 9 McCraith, et. al., 2011). During communication, individuals may interrupt others, speech may be egocentric and concrete, and they may not be able to understand the viewpoint of others (McCraith et al., 2011). New learning can occur through repetitive training, however, individuals often require minimum cognitive assistance on a daily basis to remove dangerous objects and solve any problems that occur due to changes in the environment (Allen et al., 1995; Allen et al., 1992). If clients are found to be functioning at the lower performance modes in this level, they will most likely require 24 hour assistance due to immediate problem-solving needs (Allen et al., 1995; Allen et al., 1992). Individuals functioning in the higher performance modes of this level may be able to live alone with daily checks of the environment for safety and health reasons (Allen et al., 1995; Allen et al., 1992). Allen Cognitive Level Five is titled Exploratory Actions (Allen et al., 1995; Allen et al., 1992; McCraith, et. al., 2011). Individuals functioning in Allen Cognitive Level Five understand that changes in neuromuscular control can cause different effects on objects. Allen (1999) coined the phrase neuromuscular adjustments and described this occurrence as the process that uses overt trial and error to improve the effect of actions (p. 111). Allen further elaborated that a classic example of the use of neuromuscular control is demonstrated while opening a can of paint with a screwdriver. The screwdriver acts as a primary lever and the client has to manually apply the necessary amount of pressure to gently pry open the lid. If too much force is used, the lid may fly off the container (Allen, 1999). In the lower performance modes, individuals may only consider the primary effects of their actions. In the higher performance modes, an individual considers the secondary effects of their actions and identifies the need to consult with others (Allen, 1995; Allen, 1992). At this level, individuals have a better ability to understand written and auditory information. They are able to hear and understand differences in CONTENT-ORIENTED VALIDATION 10 intonation. However, their language may appear impulsive and accusatory during difficult conversations (Allen et al., 1999). Persons functioning at this level often require supervision while learning new tasks, in order to avoid potentially hazardous situations (Allen et al., 1995; Allen et al., 1992). They may live alone with weekly checks to monitor safety and finances (Allen et al., 1995; Allen et al., 1992). In the mid-range of Level Five, individuals may also work in a job that allows for a wide margin of error or in consistent and predictable settings (Allen et al., 1995; Allen et al., 1992). In the high-range of the level, individuals may live and work independently (Allen et al., 1995; Allen et al., 1992). Allen Cognitive Level Six is titled Planned Actions (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). Clients functioning in Allen Cognitive Level Six have the ability to plan ahead and think abstractly. They are independent and do not need cognitive assistance to anticipate hazardous situations (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). Importance of identifying and understanding the Allen Cognitive Level. Allen et al. (1995; 1992) argued that it is important to determine an individuals ACL in order to recommend an environment that will maximize safety and the individuals best ability to function. Therapists should consider the client's cognitive status when developing a plan of care and making decisions with the client and caregiver about goals, treatments, and discharge planning. Effective discharge planning could include caregiver training. Caregiver training may include topics such as successful ADL and IADL completion, as well as recommendations to decrease the risk of falls, elopement, occupational deprivation, aggressive behaviors, and the proper use of verbal cues, visual cues, tactile cues, and environmental modifications during any functional task. CONTENT-ORIENTED VALIDATION 11 Cognitive programming. Several authors have developed programming based on the cognitive status of the client. Gitlin et al. (2009) designed a non-pharmacological, home-based program in which caregivers were trained to implement purposeful, prescribed activities as a means to manage behaviors during daily care. The Tailored Activities Program (TAP) is an intervention composed of four phases. In Phase I, the occupational therapist evaluated caregiver communication and management techniques and assessed the client with dementia in order to identify remaining abilities. The occupational therapist also completed an environmental assessment. In Phase II the occupational therapist educated the caregiver on the role of the environment, utilization of activities, and demonstration and practice. Phase III involved continued caregiver training in the utilization of activities. Caregivers who participated in the program noted high confidence in using the activities, decreased frustration with behavioral symptoms, and enhanced skills and personal control. The OT interventionists noted engagement and pleasure by those with dementia who received the intervention (Gitlin et al., 2009). Warchol (2004, 2006) described an interdisciplinary program for persons with dementia in long-term care. The program was based on the CDM (Allen, 1982; Allen 1985; Allen et al., 1995; Allen et al., 1992) and the theory of retrogenesis (Reisberg et al., 2002). Warchol (2004, 2006) described the need for geriatric rehabilitation specialists to utilize a comprehensive battery of performance-based cognitive assessments to identify the stage of dementia, as well as remaining functional abilities. In order to prepare healthcare professionals and caregivers to effectively interact with and care for residents with ADRD, Warchol (2004) recommended that front-line staff receive intensive training, as well as ongoing inservices. Warchol (2004, 2006) also advocated that geriatric rehabilitation professionals should consider a rehabilitative and/or CONTENT-ORIENTED VALIDATION 12 habilitative approach when providing therapy services to clients with ADRD. In addition, (Warchol, 2004, 2006) recommended that communities that serve people with ADRD (such as skilled nursing facilities, assisted living communities, and memory care communities) incorporate specialized programming for ADLs, mobility, and activities with this population. Warchol (2004, 2006) discussed the need to assess the clients ACL and incorporate a treatment approach of modification/compensation to maximize a clients remaining abilities. Geriatric rehabilitation specialists should also consider the clients internal factors, such as interests, values, and cultural considerations, as well as contextual factors, such as the environment and caregiver support when planning interventions (Warchol, 2006; Warchol, Copeland, & Ebell, 2006; Crisis Prevention Institute, 2010). In addition, Warchol (2004, 2006) argued that the clients values and support systems may directly impact treatment outcomes and maximization of the clients abilities. This is consistent with the CAN DO, WILL DO, and MAY DO, biopsychosocial perspective on function (McCraith, Austin, & Earhart, 2011, pp. 384385). As cited in McCraith et al. (2011), individuals with ADRD may experience ones best ability to function when there is a match between functional activities, activity demands, and when there is a supportive context. The CAN DO are realistic abilities, the WILL DO are relevant activities, and the MAY DO represents possible abilities based on social context and the environment (Allen & Blue, 1998). For example, a client with cognitive dysfunction may have memory impairments and experience problem-solving difficulties. When problems occur in the environment and a client lacks problem-solving skills, external sources, such as a geriatric rehabilitation specialist or caregivers can provide a spectrum of cues (Crisis Prevention Institute, 2010). These cues may range from nonspecific cues, to more specific cues, and finally to a demonstration of the solution CONTENT-ORIENTED VALIDATION 13 (Crisis Prevention Institute, 2010). The geriatric rehabilitations specialist or caregiver should observe the clients response to these cues and consider the effectiveness of the cues in terms of the clients change in task performance (Crisis Prevention Institute, 2010). If the client lacks problem solving skills, problem-solving expectations by the staff should be minimized in order to decrease the risk of frustration by the client (Crisis Prevention Institute, 2010). Cognitive functioning and learning is impacted by the individuals ability to process information (Warchol et al., 2006; Crisis Prevention Institute, 2010). When processing information, the individual must first attend to cues in the environment. If the individuals attention is not gained and maintained, information coming in through the senses will not be processed (Warchol et al., 2006; Crisis Prevention Institute, 2010). Once the individuals attention is gained, new information can be processed by working memory (Warchol et al., 2006; Crisis Prevention Institute, 2010). This new information is temporarily housed in short-term memory, may then be stored as a long-term memory, and retrieved later as a procedural memory (Warchol et al., 2006; Crisis Prevention Institute, 2010). This is similar to the informationprocessing model of memory functioning (Braungart, Braungart, & Gramet, 2011, Chapter 3). Once functional abilities and the stage of dementia are identified, interdisciplinary team members should train caregivers in order to maximize the clients functional skills and prevent unnecessary or excess disability (Warchol, 2004, 2006). The interdisciplinary team member can design a restorative or maintenance program with reasonable goals and specific approaches that caregivers can use to achieve the clients maximum functional potential on a consistent basis (Warchol, 2004, 2006). The Centers for Medicare and Medicaid Services described maintenance programs in the Medicare Benefit Policy Manual (Centers for Medicare and Medicaid Services, CONTENT-ORIENTED VALIDATION 14 2016, Ch. 15, Section 220.2, D) as an opportunity to provide patient or caregiver training in order to maintain the patients current condition or prevent the risk of functional deterioration. The use of a maintenance program can decrease the risk that a client with ADRD will experience excess disability. Excess disability is a term that describes a discrepancy between an individuals level of functioning and their actual abilities. As cited by Brody, Kleban, Lawton, and Silverman (1971), the excess disability may be greater than the medical condition warrants (Kahn, 1965). Rogers et al. (2000) identified excess disability in nursing home residents and reported that the excess disability could be alleviated if the caregivers increased opportunities for independent activity and substituted verbal assistance for physical assistance. Researchers have studied the negative effects of excess disability when a person has dementia. Excess disability has been associated with decreased walking abilities (Slaughter, Eliasziw, Morgan, & Drummond, 2011) and depression (Espiritu et al., 2001). More recently, Slaughter and Hayduck (2012) determined that the quality of the living environment is at least as important as the progression of dementia in delaying the onset of walking and eating disabilities. International Classification of Functioning, Disability and Health Model The FCA is supportive of the concepts within the International Classification of Functioning, Disability, and Health (ICF) (World Health Organization, 2002) and the CDM. The ICF (2002) is a cross-cultural, standard language and framework for health and the World Health Organizations (WHO) universal framework for health and disability. The WHO (2002) described health-related domains, such as descriptions of body function and structure, level of capacity in a standard environment, and level of performance in a usual environment. The WHO (2002) described health and functioning, rather than disability, and the model has been used as a CONTENT-ORIENTED VALIDATION 15 universal tool to measure function in society, regardless of the reason behind the impairment or disability. The WHO (2002) acknowledged that every human being can experience a certain degree of disability, which in effect, mainstreams the concept of disability. By focusing on the impact on function, users of the ICF can address the functional capacity of the person by modifying the social and physical environment (World Health Organization, 2002). The WHO developed the ICF (2002) to address equity, inclusion, and to promote the maximization of functioning in the environment. In the ICF-Model (ICF-M) (2002), the WHO emphasized a biopsychosocial approach by integrating both the medical and social models. In the medical model, disease causes a disability, which then requires a subsequent intervention to correct the disability (World Health Organization, 2002). In the social model, disability is considered to be a socially-created problem, not a problem of the person (World Health Organization, 2002). According to the WHO, neither the medical nor the social model completely explain disability nor its management (World Health Organization, 2002). According to the WHO (2002), disability and functioning are viewed as outcomes of interactions between health conditions and contextual factors, including external environmental factors, such as social attitudes, architectural characteristics, legal and social issues, and climate and terrain. Other contextual factors included internal personal factors, such as age, gender, coping styles, social background, education, profession, past and current experiences, overall behavior patterns, character, and other factors that influence the individuals disability (World Health Organization, 2002). CONTENT-ORIENTED VALIDATION 16 In the ICF-M, the WHO described performance qualifiers and capacity qualifiers (World Health Organization, 2002). Performance qualifiers are a term for what an individual does in his or her current environment. Performance was described as the lived experience (which may or may not include assistance) in the environment (World Health Organization, 2002). The capacity qualifier was a term for an individuals ability to execute a task. The capacity qualifier was the persons highest probable level of functioning at a given moment (World Health Organization, 2002). The WHO (2002) postulated that when a person has a capacity problem that is related to a health condition, that capacity problem is a component of their state of health. The WHO described that users of the ICF-M should consider both performance and capacity as they relate to participation in functional tasks (World Health Organization, 2002). Analyzing performance and capacity can help to identify the gap between actual abilities and the persons potential (World Health Organization, 2002). A gap in performance could indicate the possibility that some aspect of the environment prohibited the individual from fully utilizing his or her capabilities (World Health Organization, 2002). Assistive devices or personal assistance are one additional qualifier that can be used in this model. Assistive devices or personal assistance can not alter impairments. However, the use of assistive devices or personal assistance can minimize or remove limitations in the environment (World Health Organization, 2002). There are three underlying principles of the ICF-M: universality, parity, and neutrality (World Health Organization, 2002). The ICF-M can be universally applied to all people and their ability to function (World Health Organization, 2002). Parity refers to the notion that disability should not be referred to by etiology. Therefore, the WHO recommended that users of the ICF do not distinguish between physical and mental disabilities (World Health Organization, CONTENT-ORIENTED VALIDATION 17 2992). The language of the ICF-M has neither a positive nor a negative connotation, and is therefore considered to be neutral (World Health Organization, 2002). Similarities between the CDM and ICF-M There are significant parallels between the CDM and ICF-M. The first similarity is the emphasis on function. Claudia Allen simply stated that function is what people do! (Allen, 1999, p. 4) and that cognition impacts function. Body functions, activities, and participation are addressed in the ICF (World Health Organization, 2002). Users of both the CDM and ICF-M focus on health and functioning, as opposed to ones disability. It is important to understand an individuals cognitive abilities and maximize the individuals abilities in any given living environment with the support of care partners as needed. The demands of the environment should fit the clients cognitive abilities in order to ensure safety (Allen et al., 1995; Allen et al., 1992; McCraith et al., 2011). The notion that function can be maximized by addressing environmental or contextual barriers is explicitly addressed in the ICF-M. In the ICF-M, (World Health Organization, 2002) and CDM (Allen et al., 1995; Allen et al, 1992), it was identified that there can be a discrepancy between the individuals capacity and actual functioning due to the physical and social environment; in other words, excess disability. Allen (1999) referred to the individuals greatest ability as best ability to function (BATF), whereas, the World Health Organization (2002) referred to the individuals maximum abilities as capacity. The authors of both models identify that there may be environmental barriers and facilitators that impact an individuals ability to function (Allen, 1999; World Health Organization, 2002). CONTENT-ORIENTED VALIDATION 18 Theory of Retrogenesis The theory of retrogenesis is a reverse developmental theory in which the author concluded that the functional deficits associated with Alzheimers disease or related dementias correlate with developmental stages of humans (Reisberg et al., 2002). Reisberg et al. (2002) identified that the Functional Assessment Staging Tool cross-references stages of dementia with developmental ages, acquired abilities, lost abilities, and Alzheimers stage. Reisberg et al. (2002) recommended that caregivers should consider the stage of dementia and corresponding developmental age when providing care to the person who has ADRD. The Allen Battery of Assessments Routine Task Inventory-Expanded. In the ICF-M, standardized tools facilitate enhanced functional performance in activities through the use of universal design. There are multiple assessment tools based on the CDM that incorporate functional tasks. The Routine Task Inventory-Expanded (RTI-E) is a functional analysis of behaviors that a client exhibits during tasks such as ADLs, IADLs, communication, and work readiness (Katz, 2006). It can be completed via self-report, caregiver report, or by therapist observation and a subsequent report (Katz, 2006). The RTI-E does not provide guidelines for the use of verbal, visual, or tactile cues during task observation. Therefore, users of the RTI-E do not standardize the environment in which the functional tasks are performed. Allen Cognitive Level Screen-5 and Large Allen Cognitive Level Screen-5. Researchers have not specifically tested the psychometric properties of the Allen Cognitive Level Screen-5 (ACLS-5) or the Large Allen Cognitive Level Screen-5 (LACLS-5) CONTENT-ORIENTED VALIDATION 19 (Allen et al., 2007). However, researchers have studied the psychometric properties of earlier versions and found moderate-high levels of inter-rater reliability (Henry, Moore, Quinlivan, & Triggs, 1998; Keller & Hayes, 1998; Lee et al., 2003; Penny, Mueser, & North, 1995; Raweh & Katz, 1999; Velligan, Bow-Thomas, Mahurin, Miller, Dassori, & Erdely, 1998; Velligan, True, Lefton, Moore, & Flores, 1995). Researchers have reported significant test-retest reliability of the ACLS (McAnama, Rogosin-Rose, Scott, Joffe, & Kelner, 1999). Researchers have also reported positive correlations between ACLS and LACLS scores and scores from measures of ADLs and IADLs (Keller & Hayes, 1998; Mcanama et al., 1999; Velligan et al., 1998; Velligan et al., 1995; Ziv, Roitman, & Katz, 1999); living situation (Henry et al., 1999; McAnanama et al., 1999), and social competence (Penny et al., 1995). Allen Diagnostic Module-2 Edition. nd If a potential cognitive impairment is identified on the ACLS-5 or LACLS-5, the therapist can choose an assessment from the Allen Diagnostic Module-2 Edition (ADM-2) nd (Earhart, 2006) to verify a cognitive impairment or measure changes in functional cognition. Over 25 novel, craft-based performance assessments from the ADM-2 are intended to be used to assess functional cognition within the context of the cognitive disabilities model. The ADM was designed for standardized administration and has accompanying scoring rubrics (Earhart, 2006). However, critics of the ADM-2 argue that it is difficult to accurately ascertain ones ability to function in his or her environment from performance on a standardized craft project. CONTENT-ORIENTED VALIDATION 20 Other assessments that are functional in nature. Other standardized assessments are described in the literature. The Cognitive Performance Tool (CPT) is a standardized, performance-based test of ADLs and IADLs (Burns, 2006). The CPT was grounded in the cognitive disabilities model and designed to assess a functional level in clients who have Alzheimers disease (Burns, 1992). However, the current version of the CPT corresponds with the cognitive disabilities reconsidered model (Levy & Burns, 2011, Chapter 18). Users of the CPT assess functional cognition during the seven subtasks, which include: (a) Medbox, (b) Shop, (c) Phone, (d) Travel, (e) Toast, (f) Wash, and (g) Dress. The scores on the CPT are associated with six profiles that correspond to levels in the cognitive disabilities reconsidered model (Levy & Burns, 2011). In a recent inter-rater reliability study, it was found that the results of the CPT provide users with accurate and consistent information (Schaber, Stallings, Brogan, & Ali, 2016). Test users should be cautioned, however, that the cognitive disabilities model reconsidered is not interchangeable with the original CDM. Therefore, occupational therapists using the CPT should not use the CDMs scoring and interpretation (McCraith et al., 2011). The Executive Function Performance Test (EFPT) is standardized, performance-based test of IADLs (Baum, Morrison, Hahn, & Edwards, 2007). The EFPT was developed using the person-environment-occupational performance model (Baum, et al., 2008). The purpose of using the EFPT is to identify impairments in executive function, determine capacity for executive function, and determine the amount of assistance required during the executive function tasks (Baum et al., 2007). The subtests of the EFPT include (a) preparing or heating up a light meal, (b) managing medications, (c) using the telephone, and (d) paying bills. Researchers have demonstrated that the EFPT is a valid and reliable tool and there is CONTENT-ORIENTED VALIDATION 21 support for the inter-rater reliability and construct, criterion, and discriminant validity with clients who have experienced a mild or moderate stroke (Baum, et al., 2008). People with multiple sclerosis performed significantly worse on the EFPT than healthy people (Goverover et al., 2005). In a study involving people with schizophrenia and use of the EFPT, internal consistency reliability was high, construct validity was significant, and there was moderate to high criterion validity (Katz, Tadmor, Felzen, & Hartman-Maeir, 2007). The Alternate EFPT (aEFPT) was developed to provide occupational therapists with additional testing tasks. No statistically significant differences were found between the EFPT and the aEFPT. Therefore, the aEFPT was found to be comparable to the EFPT to identify performance deficits in clients who have experienced a stroke (Hahn et al., 2014). Traditionally, OT practitioners have been asked to determine a persons capacity to live at home safely, work, and engage in valued occupations (Baum et al., 2008). While the CPT, and EFPT are functional in nature, they are not theoretically based on the cognitive disabilities model. The RTI-E was not designed to incorporate specific cueing protocols for optimal administration to clients with ADRD. As a result, there is a substantial need for a standardized, performance-based assessment grounded within the cognitive disabilities model to guide cueing and scoring of skilled observations during ADLs, IADLs, and leisure tasks with clients who have ADRD. The Functional Cognitive Assessment The FCA was developed to measure the construct of functional cognition in adults with ADRD through assessing how the client performs everyday tasks (Ebell et al., 2016). The FCA was developed using the CDM, however, it inherently was also influenced by the theory of CONTENT-ORIENTED VALIDATION 22 retrogenesis (Reisberg et al., 2002) and has parallels with the ICF-M (World Health Organization, 2002). The FCA has standardized administration and scoring guidelines. First, test administrators determine if the associated tasks are familiar or unfamiliar to the client (Ebell et al., 2016). A client may demonstrate optimal abilities during tasks that are familiar, meaningful, relevant, or which engage procedural memories. Alternately, a client may have the opportunity to demonstrate new learning skills and other executive function skills while completing unfamiliar tasks or the client may not perform optimally if a task is considered to be irrelevant or unfamiliar (Ebell et al., 2016). During the test, the therapists standardized use of cues to prompt performance corresponds with a standardized environment in the ICF-M. Because sensory cues are gradually added as needed during each task, the results of the test are representative of the clients best ability to function, which is described as the capacity qualifier in the ICF-M. Allen et al. (2007) emphasized the importance of assessing the clients use of sensory cues to complete motor actions during a cognitive assessment. Users of the FCA identify the type and amount of verbal, visual, and tactile cues that the client requires during a variety of functional tasks (four ADLs, three IADLs, and three leisure tasks) (Ebell et al., 2016). The graded cues that may be provided by the therapist are for the purpose of gaining and maintaining the clients attention to the given tasks and for identifying and solving problems, in order to elicit a positive response or change in behavior during task completion (Ebell et al., 2016). (See Appendix A) Analysis is needed to establish evidence of content-oriented validity and determine if the FCA can be used to identify patterns of behaviors that are associated with ACLs. CONTENT-ORIENTED VALIDATION 23 Content-Oriented Validity Evidence As cited in the APA Handbook of Testing and Assessment in Psychology (2013, vol. 1, p.12), involving subject matter experts is one way to evaluate if a tests content is indicative of the construct that is being measured and if the test items are relevant and representative of the construct (Haynes, Richard, & Kubany, 1995). According to the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014), Test content refers to the themes, wording, and format of the items, tasks, or questions on a test (p.14). In addition, inter-rater agreement amongst the subject matter experts should be explicitly stated (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014, p. 25). The peer-reviewed literature describes both quantitative and qualitative methods for establishing content-oriented validity evidence. Several methods for analyzing content-oriented validity evidence from a quantitative perspective can be located in the literature. Quantitative methods include calculation of the interrater agreement (IRA), content validity index (CVI), and factorial validity index (FVI) (Rubio, Berg-Weber, Tebb, Lee, & Rauch, 2003). A variety of other researchers have also discussed the use of the CVI in instrument development (Claeys, Neve, Tulkens, & Spinewine, 2012; Delgado-Rico, Carretero-Dios, & Ruch, 2012; DeVon et al, 2007; Lynn, 1986; Malmgreen, Graham, Shortridge-Baggett, L. M., Courtney, M., & Walsh, 2009; Polit & Beck, 2006; Polit, Beck, & Owen, 2007). Researchers have acknowledged that one weakness of the CVI is the failure to adjust for chance agreement amongst the reviewers. Therefore, Wynd, Schmidt, and Schaefer (2003) and Polit, Beck, and Owen (2007) recommended translating itemlevel CVIs (I-CVIs) into values of a modified kappa statistic. CONTENT-ORIENTED VALIDATION 24 Methods Research Design This study used a cross-sectional design using a web-based survey to establish contentoriented validation of the FCA. A cross-sectional design was selected in order to receive feedback about the representativeness and clarity of the items during the evaluation phase of the FCA. According to Rubio et al. (2003), if researchers do not complete a content validity study, they would risk disseminating an untested measure to clinicians for a pilot study. If recommendations were provided after the pilot study and test revisions occurred as a result of the pilot study, the researchers would need to again pilot the test with another subject pool (Rubio et al., 2003). Therefore, a content validity study can save valuable resources by analyzing the measure prior to the pilot phase (Rubio et al., 2003). Sampling The Human Protections Administrator for the University of Indianapolis determined that the proposal was not eligible for Institutional Review Board (IRB) review because the study was not within the purview of human research protections. The Human Protections Administrator determined that the proposal did not meet the definition of human subjects research as set forth in the federal regulations at 45 CFR 46.102 (U.S. Department of Health and Human Services, 2009) because the information being elicited concerned the FCA, and not about the respondent. Please see Appendix B for the letter of formal certification of institutional review and determination. The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on CONTENT-ORIENTED VALIDATION 25 Measurement in Education, 2014) described that content-related evidence can come from expert judges and recommended that test developers fully describe the procedures used to select the experts and that the qualifications and experience of the judges should be presented. For the purposes of this research study, professional experts were defined as individuals with at least an associates degree who have published papers or presented at state, regional, or national meeting on the topic of implementation of the cognitive disabilities model with clients who have ADRD or have experience in psychometric testing. Recruits with experience in psychometric testing were included in the inclusion criteria because they would possibly be able to provide additional information regarding test construction (Davis, 1997). Rehabilitation professionals who did not meet the above criteria were considered for inclusion as lay experts if they had at least five years of work experience using the cognitive disabilities model with older adults who have ADRD. The respondents were asked to provide their professional opinion regarding relevance and clarity of the tests content as compared to the construct of functional cognition. The criteria for professional and lay experts was described in a survey question. Research participants had the option to identify as either a professional or lay expert on the survey. Rubio et al. (2003) recommended using three to 10 professional experts and three to 10 lay experts yielding a possible sample size of six to 20 experts. Polit, Beck, and Owen (2007) recommended that the first round of expert content validation would ideally have a large panel, such as eight to 12 experts. Polit, et al. (2007) developed a table for three to nine content experts that adjusted for the possibility of chance agreement amongst experts, in order to determine fair, good, or excellent Item Level-Content Validity Index (I-CVI). Therefore, this researcher attempted to recruit nine professional experts and nine lay experts to participate in this content CONTENT-ORIENTED VALIDATION 26 validity study. The survey was sent to 33 individuals who potentially met the qualifications for a professional or lay expert. The participant recruitment email was comprised of the informed consent for online data, which included contact information for the researchers, the purpose of the study, inclusion criteria for the participants, a description of The FCA, an explanation of the survey, and an active hyperlink for the survey. Proceeding with the survey indicated consent. Participation in the study was completed anonymously; however, some recruits and respondents personally contacted this author to discuss this research study. Follow up reminder emails were sent weekly. The informed consent for online data collection and the active hyperlink were sent to participants four times in order to give the respondents adequate time to complete the survey. Rubio et al., (2003) recommended offering the content experts a final version of the scale as an incentive to participate in the content validity study. Upon conclusion of this survey, the respondents had the option to click on an active hyperlink, which routed them to another survey. If they so desired, respondents entered their contact information in order to receive a free copy of the FCA after the conclusion of the study and after the test developers revised the FCA. Instrumentation Data were collected via a Qualtrics survey delivered to respondents via electronic mail. Within the Qualtrics survey (Appendix C), the respondents evaluated four topics for each task of The Functional Cognitive Assessment: (a) representativeness of the content domain, (b) clarity of the item, (c) factor structure, and (d) comprehensiveness. Representativeness described an items ability to represent the content domain of functional cognition. The clarity of an item CONTENT-ORIENTED VALIDATION 27 was evaluated on how clearly the item was worded. Respondents were also asked to assign each task to a factor (area of occupation). Pilot Study Prior to submitting the proposal to the IRB, the survey was piloted with five individuals. All five individuals worked in the rehabilitative field of occupational therapy, physical therapy, or speech-language pathology. Two individuals offered specific suggestions to enhance the usability of the survey. Based on feedback from the pilot study, changes were made to enhance the clarity of the introduction and to include additional context on test administration. In addition, the font size of the Qualtrics survey questions was increased and minor revisions related to punctuation and formatting were made to the survey. One respondent in the pilot study noted that it might take the actual study participants approximately 30 minutes to complete the survey. The Survey The survey included a definition of the construct of functional cognition, an opportunity for the expert to identify as a professional or lay expert, initial survey instructions, and the standardized test instructions that test users are recommended to state prior to administering a task to a client. Throughout the survey, respondents were asked to rate the representativeness of the content domain (ability of the test item to represent the content domain as established in the theoretical definition of functional cognition), rate the clarity of each test item (how clearly an item is worded), to select the factor (area of occupation) associated with each test item, and to rate the comprehensiveness of the FCA. CONTENT-ORIENTED VALIDATION 28 The respondents evaluated the description of each task and scoring rubric on representativeness and clarity using a scale of one through four. Anchors were provided for the scale points. For example, a value of one indicated that the item was not representative of the domain or was not clear. A value of four indicated that the item was representative of the domain or clear. As cited by Rubio et al. (2003) it is recommended to use a four-point scale, in order to prevent content experts from choosing the middle score if they are unsure of a response (Lynn, 1986). For representativeness, the scoring criteria were as follows: (a) 1=Item is not representative of functional cognition; (b) 2=Item needs major revisions to be representative of functional cognition; (c) 3=Item needs minor revisions to be representative of functional cognition; (d) 4=Item is representative of functional cognition; and (e) I prefer not to answer. Respondents were next asked to make any comments about the representativeness of the task. For clarity, the scoring criteria were as follows: (a) 1=Item is not clear; (b) 2=Item needs major revisions to be clear; (c) 3=Item needs minor revisions to be clear; (d) 4=Item is clear, and (e) I prefer not to answer. Respondents were then asked to indicate any comments regarding the clarity of the task instructions and scoring rubric. The respondents were asked to assign each test item to a factor. Several factors were listed for the construct of functional cognition. These factors were selected from OTPF-III. (American Occupational Therapy Association, 2014). The factors were: (a) ADLs, (b) IADLs, (c) leisure tasks, or (d) other. Respondents had the opportunity to make comments regarding the factor that was selected for the task in terms of its fit. The comments section enabled the respondents to identify a different occupation that was not listed or whether the task encompassed multiple occupations. Finally, the respondents were asked to address the comprehensiveness and CONTENT-ORIENTED VALIDATION 29 thoroughness of the entire assessment by recommending which items should be deleted or added to the assessment. The format of the survey was modeled after Grant and Daviss (1997) and Rubio et al.s (2003) recommendations for structural elements that should be included within content reviews. Throughout the survey, items were designed as forced-response questions. However, respondents had the option of selecting I prefer not to answer in order to provide the opportunity to avoid a question and still proceed through the survey. Plan for Data Analysis Data were exported and analyzed using Microsoft Excel. As recommended by Rubio et al. (2003), four types of analyses were performed: (a) IRA for representativeness and clarity, (b) CVI for representativeness, (c) FVI, and (d) I-CVI using values of a modified kappa statistic. Polit, Beck, and Owen (2007) reported that a weakness of the CVI is the failure to adjust for chance agreement. Therefore, Polit et al. (2007) recommended translating I-CVIs into values of a modified kappa statistic. The probability of chance and modified kappa statistic were calculated for representativeness and clarity of each test item. This approach is recommended for studies that have more than five experts because as the number of experts increases, the chances of all of them agreeing decreases (Lynn, 1986). Interrater agreement. Rubio et al. (2003) described a method to quantify content validity, which included the use of the interrater agreement (IRA), content validity index (CVI), and the factorial validity index (FVI). The IRA determines the extent to which the experts agree that the item is representative of the construct and is clearly written. Rubio et al. recommended (2003) calculating the IRA for representativeness and clarity for each test item and the overall test in CONTENT-ORIENTED VALIDATION 30 order to assess the extent to which the experts are reliable in their ratings. Rubio et al. (2003) advocated that the researcher can count the number of test items rated as one or two on the scale and the items rated three or four on the scale. The IRA for each item can be calculated by determining the agreement between experts. The IRA for the entire scale can also be calculated by counting the number of items that have an IRA of at least 0.80 and dividing that number by the total number of items. Rubio et al. (2006) did not offer a recommendation regarding an acceptable IRA. Rubio et al. (2003) recommended that the IRA be calculated for representativeness and clarity in order to assess the extent to which the experts are reliable in their ratings. The scale is dichotomized, with values of one and two combined and values of three and four combined. The researcher counts the items that the experts rated as one or two and three or four. As the number of experts increases above five, a conservative approach is recommended due to the decreased likelihood that the experts will all agree. A conservative approach for IRA can be calculated by counting the number of items that have an IRA of at least .80 and dividing that number by the total number of items (Lynn, 1986; Rubio et al., 2003). Lynn (1986) created a table to show the proportion of experts whose endorsement of an item or instrument is required to establish content validity beyond the .05 level of significance. With the use of 10 experts, the recommendation is .78 or greater. Content validity index. The CVI of a tool can be calculated based on the representativeness of the measure to the construct or items being measured. Rubio et al. (2003) recommended that the CVI be calculated for the representativeness of each item by counting the number of experts who rated the item as CONTENT-ORIENTED VALIDATION 31 three or four and dividing that number by the total number of experts. This yields the proportion of experts who deemed the item as content valid. For a study with six to 10 experts, Lynn (1986) recommended a minimum item CVI of .78. Polit and Beck (2006) referred to this as item-level CVI (I-CVI). As referenced by Polit and Beck (2006) I-CVI should be no lower than .78 when there are six or more judges (Lynn, 1986). Researchers can also investigate the CVI for the entire measure. Rubio et al., (2003) recommended that the CVI for the measure be estimated by calculating the average CVI across the items. It has been recommended that new measures have an overall CVI of at least .80 (Davis, 1992). Polit and Beck (2006) referred to this as Scale-CVI Average (S-CVI/Ave). The S-CVI/Ave can be estimated by calculating the average CVI across the items. Polit, Beck, and Owen (2007) recommend that a scale should have S-CVI/Ave of .90 or higher to indicate excellent content validity. Polit et al. (2007) argued that this stringent S-CVI/Ave adjusts for chance agreement amongst the experts and that there is strong conceptual work, good items, outstanding subject matter experts, and that there were clear instructions to the experts. Factor validity index. Rubio et al. (2003) created the FVI to determine the degree to which the experts appropriately assigned the items to their respective factors. The factors of the FCA included categories for (a) ADLs, (b) IADLs, (c) leisure, and (d) other. The number of experts who correctly assigned the item with the factor was divided by the total number of experts. The average was calculated across items to compute the FVI for the measure. Rubio et al. (2003) recommended an FVI of at least .80. CONTENT-ORIENTED VALIDATION 32 Item-content validity index using a modified kappa statistic. Wynd, Schmidt, and Schaefer (2003) and Polit et al. (2007) recommended translating item-level CVIs (I-CVIs) into values of a modified kappa statistic (k*). This technique was recommended to adjust for the possibility of chance agreement amongst expert raters. First, the probability of chance (Pc) agreement was computed using the formula for a binomial random variable. Next, the modified kappa (k*) statistic was calculated for the representativeness of each item. Cicchetti and Sparrow (1981) and Fleiss (1981) described evaluation criteria as follows for kappa: Fair=k of .40 to .59; Good=k of .60-.74; and Excellent = k>.74. Results Fifteen respondents participated in the survey, which yielded a 45% response rate. Due to an unclear submit icon on the survey, data were not saved for multiple participants. One respondent only answered one question, in which the respondent free-typed a response. Therefore, data from 10 participants were considered in the data analysis, which yielded a final response rate of 30%. Based on the date of when it was identified that data were not being saved and which respondents had entered their contact information to obtain a final version of the FCA, this author contacted those experts and notified them that it was possible that their research data were not saved. Several respondents sent their impressions of the survey and subjective feedback to this researcher. Their comments were considered during the revision stage of the test development. However, the raw data was not included in this data analysis. Interrater agreement The interrater agreement (IRA) was calculated for representativeness and clarity of each task (see Table 1). The IRA for representativeness of each item ranged from 0.70-0.90. The CONTENT-ORIENTED VALIDATION 33 Medication Management task did not meet the 0.80 criteria for acceptability. This yielded a scale IRA of 0.90 for representativeness, which is acceptable for a scale IRA and exceeds the recommendation by Lynn (1986) and Polit et al. (2007) of 0.78. The IRA for clarity of each item ranged from 0.60-0.90. Making seasoned rice, washing dishes, and medication management were below the 0.80 recommendation. Therefore, the scale IRA for clarity was of 0.70, which is below the recommended level of 0.78 by Lynn (1986) and Polit et al. (2007). Content Validity Index (representativeness) The CVI of the FCA was calculated for representativeness of each task, as well as the entire measure (see Table 1). The CVI for representativeness ranged from 0.9 for the bathing task, to 0.8 (dressing, oral care, making seasoned rice, making coffee, washing dishes, playing cards, gardening, and using a remote control), to 0.7 for the task of medication management. Since the task of medication management task was below the 0.78 cutoff criteria, the medication management task was removed from the calculation. Therefore, the S-CVI Average was 0.81 for representativeness. This is above the recommended S-CVI Average of at least 0.8 for new measures (Davis, 1992; Polit et al., 2007). Factor Validity Index The FVI was 1.0 for the tasks of bathing, dressing, making seasoned rice, making coffee, washing dishes, medication management, playing cards, and gardening (see Table 1). The FVI was .9 for oral care and using a remote control. The average FVI for the FCA was 0.98. The FVI for the FCA exceeded the recommended FVI of 0.8 (Rubio et al. 2003). CONTENT-ORIENTED VALIDATION 34 I-CVI Using a Modified Kappa Statistic The modified kappa statistics for representativeness (see Table 1) ranged from 0.90 (Bathing), 0.79 (Dressing, Oral Care, Making Seasoned Rice, Making Coffee, Washing Dishes, Playing Cards, Gardening, and Using a Remote Control), to 0.66 (Medication Management). All of the tasks for representativeness were evaluated as being excellent, with the exception of Medication Management, which was rated as good. Table 1 Data Analysis for Content-Oriented Validation of the Functional Cognitive Assessment IRA CVI Rep. Clarity Rep. Bathing 0.90 0.90 0.90 Dressing 0.80 0.80 Oral Care 0.80 Making Seasoned Rice FVI I-CVI Rep. Rating Pc k* 1 0.01 0.90 Excellent 0.80 1 0.05 0.80 Excellent 0.80 0.80 0.90 0.05 0.80 Excellent 0.80 0.60 0.80 1 0.05 0.80 Excellent Making Coffee 0.80 0.90 0.80 1 0.05 0.80 Excellent Washing Dishes 0.8 0.70 0.80 1 0.05 0.80 Excellent Medication Management 0.70 0.60 0.70 1 0.12 0.66 Good CONTENT-ORIENTED VALIDATION 35 Playing Cards 0.80 0.80 0.80 1 0.05 0.80 Excellent Gardening 0.80 0.80 0.80 1 0.05 0.80 Excellent Using a Remote Control 0.80 0.80 0.80 0.9 0.05 0.80 Excellent Scale Scale Scale Scale 0.90 0.70 0.81 0.98 Respondent Feedback The respondents provided some useful comments for revisions of the FCA. Particularly helpful recommendations included the need to clarify the initial standardized instructions in the introduction of the test. In addition, the respondents requested further clarification of the task segmentation when verbal, visual, or tactile cues are introduced during the standardized administration procedures of each task. The respondents provided very specific feedback about the scoring rubrics. For example, one respondent identified the need to switch the scoring rubrics from an active voice to a passive voice. The need to enhance the descriptions of observed abilities in the scoring rubrics for Allen Cognitive Levels one and two was identified. Regarding the medication management task, it was noted that there was a need to expand upon the scoring rubric in order to include a more comprehensive task analysis of medication management. It was also indicated that a revision of the title of the medication management task might better reflect the observations noted in the scoring rubric. CONTENT-ORIENTED VALIDATION 36 Specific feedback regarding the medication management task. The respondents had useful comments about the medication management task, such as, This assessment is looking at one aspect of medication management. It does not appear to assess the client's awareness of how to order medications, when the medications are scheduled, why the medications are prescribed, or what conditions the medications are treating. . . The task being assessed here is "taking medications." Similarly, another respondent wrote, The domain being assessed, the directions and the rubric do not align . . . medication management involves more than the act of taking pills. The assessment does not appear to assess medication management in its entirety. The rubric indicates behaviors that a client may demonstrate at a particular cognitive level, however the assessment does not explore these potential behaviors. Another respondent wrote, Would suggest revising Level 1-2 since clearly medication would be administered to a person functioning at this level, and the observation of responses to being provided...medication is the behavior to be observed. . . this isn't clear since the instructions prompt the rater to have the person take the medication themselves. In terms of the clarity of the medication management task, one respondent noted that for Low Level 3, the last bullet point in the scoring rubric could be misinterpreted that the caregiver should consume the medication. Discussion The purpose of this study was to establish content-oriented validity evidence of the FCA, which is a necessary step during test development. Based on the results of the data, there is content-oriented validity evidence for the FCA. A group of professional experts objectively evaluated the tool to determine the representativeness, clarity of the items, the area of occupation CONTENT-ORIENTED VALIDATION 37 with which the items is associated, and if any test items should be added or deleted. The tasks of bathing, dressing, oral care, making seasoned rice, making coffee, washing dishes, playing cards, gardening, and using a remote control were rated as having excellent representativeness of the construct of functional cognition. The task of medication management was rated as having good representativeness of the construct of functional cognition. FVI was well above the recommended threshold of 0.8. Based on the results of the survey, it was determined that a variety of occupations identified on OTPF-III (2014) are included in the FCA. Tasks of the FCA, encompass occupations such as ADLs, IADLs, and leisure tasks. Recommendations gathered from this study were utilized to clarify the assessment and minor revisions were made to the administration and scoring manual. Based on feedback from a respondent, it is now permissible to consider asking the client to select which testing tasks are preferred in order to capitalize on the clients motivation during the test. The test administrators instructions for all 10 tasks were revised in order to clarify the general instructions, include the clients preferences during task completion, answer any questions that the client might have during task completion, and further describe the use of verbal cues by the test administrator. The IRA for clarity was below Polit and Becks (2006) recommendation of 0.78 for the tasks of making seasoned rice, washing dishes, and medication management. Therefore, it is recommended that those items be revised in order to enhance the clarity of each task. The CVI for representativeness of the medication management task was 0.70. Based on the recommendation by Polit and Beck (2006), items with CVI <.78 are candidates for revision. Therefore, it is recommended that the task of medication management be revised. It is also recommended that the medication task is renamed for clarity. Perhaps by changing the title from Medication Management to Taking Medications. This title change would better reflect the CONTENT-ORIENTED VALIDATION 38 observations that were noted in the scoring rubric. It is further recommended specific pill boxes are included in the task of Taking Medications. Finally, it is recommended that aspects of overall medication management such as renewing prescriptions, taking the medications at the prescribed time of day, and consuming the medications be removed from the scoring rubric. The CVI-Scale and FVI for the FCA were considered to be strong. Test items within the FCA are categorized in the following areas of occupation (a) Activities of Daily Living, (b) Instrumental Activities of Daily Living, and (c) Leisure. The excellent FVI and identification of additional areas of occupation further supports the notion that the FCA is a comprehensive tool which can be used to assess functional cognition in clients who have ADRD. There are no occupations related to work or social participation in the FCA. Clients with ADRD may participate in these occupations. In order to address additional, relevant occupations, it is recommended that work and social participation tasks be included in the FCA. Potential Use of the FCA The FCA could be used to confirm core clinical criteria associated with the diagnosis of dementia. For example, cognitive symptoms must be severe enough that they interfere with work or usual activities (McKhann et al., 2011). The client must also have an impaired ability to acquire and remember new information, impaired reasoning and judgement, impaired visuospatial abilities, and impaired language (McKhann et al., 2011). To diagnose a client with probable Alzheimers Disease dementia, the client must meet the criteria for dementia, demonstrate an insidious onset, and worsening of cognitive symptoms by report or observation McKhann et al., 2011). Improper diagnosis of a disorder may lead to inappropriate or harmful treatments (Sireci & Sukin, 2014). If a healthcare provider has concerns that a client is CONTENT-ORIENTED VALIDATION 39 exhibiting the core clinical criteria associated with the diagnosis of dementia, the client could be referred to an OT practitioner who is trained to administer and interpret the FCA within the context of the CDM. The OT practitioner could confirm or deny the presence of the core clinical criteria associated with dementia. Once evidence of reliability is established for the FCA, the FCA would be useful to identify and describe the impact of dementia on functional cognition as well as track changes over time. Limitations All of the respondents identified themselves as professional experts. Therefore, there were no lay experts in the study. However, it should be noted that the professional experts worked in both academic and clinical settings. In addition, not all of the respondents responses were saved on the survey due to the unclear submit icon at the end of the survey. Because the responses were not submitted, they were not able to be recovered. It is also important to caution readers that this study was only designed to establish content-oriented validity evidence. Other aspects of validity evidence, such as construct validity, criterion validity, concurrent validity, and predictive validity have not been investigated, nor has reliability. Pending revisions following this initial content validity study, it is recommended that another content validity study occur in order to assess the revised test (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). Future research should also investigate other aspects of validity, reliability, and pilot testing of the FCA. Studies should include culturally diverse populations in order to determine if the FCA can be effectively used with a variety of clients with different backgrounds. CONTENT-ORIENTED VALIDATION 40 Conclusion Administration of the FCA could be used to identify the impact of ADRD during the completion of ADL, IADL, and leisure tasks. When OT practitioners utilize the FCA with clients who have ADRD, they can share the results and subsequent recommendations with caregivers about safety and the clients best ability to function. Occupational therapy practitioners can specifically inform caregivers about the verbal, visual and/or tactile cues that are useful to maximize the clients performance; thereby, decreasing the risk of excess disability, decreasing the risk of potential safety hazards, and reducing the risk of hospital readmissions. CONTENT-ORIENTED VALIDATION 41 References Allen, C. K. (1982). Independence through activity: The practice of occupational therapy (psychiatry). American Journal of Occupational Therapy, 36, 731-739. Allen, C. K. (1999). Structures of the Cognitive Performance Modes. Ormond Beach, FL: Allen Conferences. Allen, C. K., Austin, S. L., David, S. K., Earhart, C. A., McCraith, D. B., & Riska-Williams, L. (2007). Manual for the Allen Cognitive Level Screen-5 (ACLS-5) and Large Allen Cognitive Level Screen-5 (LACLS-5). Camarillo, CA: ACLS and LACLS Committee. Allen, C. & Blue, T. (1998). Cognitive disabilities model: How to make clinical judgments. In N. Katz (Ed.), Cognition and occupation in rehabilitation: Cognitive models for intervention in occupational therapy (pp. 225-279). Rockville, MD: American Occupational Therapy Association. Allen, C. K., Blue, T., & Earhart, C. A. (1995). Understanding cognitive performance modes. Ormond Beach, FL: Allen Conferences. Allen, C. K., Earhart, C. A., & Blue, T. (1992). Occupational therapy treatment goals for the physically and cognitively disabled. Bethesda, MD: The American Occupational Therapy Association, Inc. Alzheimers Association. (2012). 2012 Alzheimers disease facts and figures: Includes a special report on people with Alzheimers disease and other dementias who live CONTENT-ORIENTED VALIDATION 42 alone. Alzheimers disease, 8(2). Retrieved from https://www.alz.org/downloads/facts_figures_2012.pdf Alzheimers Association. (2015). 2015 Alzheimers disease facts and figures: Includes a special report on disclosing a diagnosis of Alzheimers disease. Alzheimers & Dementia, 11(3), 332+. Retrieved from https://www.alz.org/facts/downloads/facts_figures_2015.pdf Alzheimers Association. (2016). 2016 Alzheimers disease facts and figures: Includes a special report on the personal financial impact of Alzheimers on families. Alzheimers and Dementia, 12(4), 1-84. Retrieved from https://www.alz.org/documents_custom/2016facts-and-figures.pdf American Educational Research Education, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Education. American Occupational Therapy Association. (2013). Cognition, Cognitive Rehabilitation, and Occupational Performance. American Journal of Occupational Therapy, 67, S9S31. doi: 10.5014/ajot.2013.67S9 American Occupational Therapy Association. (2014). Occupational therapy practice framework: Domain and process (3rd ed.). American Journal of Occupational Therapy, 68(Suppl.1), S1-S48. http://doi.org/10.5014/ajot.2014.682006 CONTENT-ORIENTED VALIDATION 43 American Occupational Therapy Association. (2016). Role of occupational therapy in assessing functional cognition. Retrieved from http://www.aota.org/advocacy-policy/federal-regaffairs/resources/role-ot-assessing-functional-cognition.aspx Apostolova, L. G. & Cummings, J. L. (2008). Neuropsychiatric manifestations in mild cognitive impairment: A systematic review of the literature. Dementia and Geriatric Cognitive Disorder, 25(2), 115126. doi: 10.1159/000112509 Baum, C. M., Morrison, T., Hahn, M., & Edwards, D. F. (2007). Test protocol booklet: Executive Functional Performance Test. Retrieved from http://www.ot.wustl.edu/about/resources/executive-function-performance-test-efpt308 Baum, C. M., Connor, L.T., Morrison, T. Hahn, M., Dromerick, A. W., Edwards, D. F. (2008). Reliability, validity, and clinical utility, of the Executive Function Performance Test: A measure of executive function in a sample of people with stroke. American Journal of Occupational Therapy, 62, 446-455. Braungart, M. M., Braungart, R. G., & Gramet, P. R. (2011). Applying learning theories to healthcare practice. In Bastable, S.B., Gramet, P., Jacobs, K., & Sopczyk, D.L., Health Professional as Educator (pp. 63-110). Sudbury, MA: Jones & Bartlett Learning. Brody, E. M., Kleban, M. H., Lawton, M. P., & Silverman, H. A. (1971). Excess disabilities of the mentally impaired aged: Impact of individualized treatment. Gerontologist, 11, 124-133. Burns, T. (2006). Cognitive Performance Test (CPT). Pequannock, NJ: Maddak. CONTENT-ORIENTED VALIDATION 44 Bynum, J. (2011). Unpublished tabulations based on data from the Medicare current beneficiary survey for 2008. Prepared under contract by Julie Bynum, MD, MPH, Dartmouth Institute for Health Policy and Clinical Care, Dartmouth Medical School. Centers for Medicare and Medicaid Services. (2016). Medicare Benefit Policy Manual. Retrieved from https://www.cms.gov/Regulationsand- Guidance/Guidance/Manuals/Downloads/bp102c15.pdf Claeys, C., Neve, J., Tulkens, P. M., & Spinewine, A. (2012). Content validity and interrater reliability of an instrument to characterize unintentional medication discrepancies. Drugs Aging, 29(7), 577-591. Crisis Prevention Institute. (2010). Dementia Capable Care: Dementia Therapy Intermediate. Dementia Care Specialists: a CPI specialized offering. Milwaukee, WI: Crisis Prevention Institute. Davis, L. (1992). Instrument review: Getting the most from your panel of experts. Applied Nursing Research, 5, 194-197. Davydow, D. S., Zibin, K., Katon, W. J., Pontone, G. M., Chwastiak, L., Langa, K. M., & Iwashyna, T. J. (2014). Neuropshychiatric disorders and potentially preventable hospitalizations in a prospective cohort study of older Americans. Journal of General Internal Medicine, 29(10), 1362-1371. Delgado-Rico, E., Carretero-Dios, H., & Ruch, W. (2012). Content validity evidences in test development: An applied perspective in test development. Journal of Clinical and Health Psychology, 12(3), 449-460. CONTENT-ORIENTED VALIDATION 45 DeVon, H. A., Block, M. E., Moyle-Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J., Savoy, S. M., & Kostas-Polston, E. (2007). A Psychometric Toolbox for Testing Validity and Reliability. Journal of Nursing Scholarship, 39(2), 155-164. Doucet, B. M., Woodson, A., & Watford, M. (2014). Centennial VisionMoving toward 2017: Progress in rehabilitation intervention effectiveness research. American Journal of Occupational Therapy, 68, e124-e148. http://dx.doi.org/10.5014/ajot.2014.011874 Dunn, W., McClain, L. H., Brown, C., & Youngstrom, M. J. (1998). The ecology of human performance. In M. E. Neistadt and E. B. Crepeau (Eds.), Willard and Spackmans Occupational Therapy (9 ed., pp. 525-535). Philadelphia: Lippincott th Williams and Wilkins. Earhart, C. A. (2006). Allen Diagnostic Module-2 Ed. Colchester, CN: S & S Worldwide. nd Ebell, C., Ford, A. A. & Warchol, K. (2016). Standardized Administration and Scoring Manual for the Functional Cognitive Assessment. Milwaukee, WI: Crisis Prevention Institute. Elliot, A. F., Burgio, L. D., & DeCoster, J. (2010). Enhancing caregiver health: Findings from the Resources for Enhancing Alzheimers caregiver Health II Intervention. Journal of the American Geriatrics Society, 58(1), 30-37. doi: 10.1111/j.15325415.2009.02631.x Espiritu, D. A. V., Rashid, H., Mast, B. T., Fitzgerald, J., Steinberg, J., & Lichtneberg, P. A. (2001). Depression, cognitive impairment and function in Alzheimers disease. International Journal of Geriatric Psychiatry, 16, 1098-1103. CONTENT-ORIENTED VALIDATION 46 Feng, Z., Coots, L. A., Kaganova, Y., & Wioner, J. M. (2014). Hospital and ED use among Medicare beneficiaries with dementia varies by setting and proximity to death. Health Affairs, 33(4): 683-690. doi: 10.1377/hlthaff.2013.1179 Gitlin, L. N., Winter, L., Earland, T. V., Herge, E. A., Chernett, N. L., Piersol, C. V., & Burke, J.P. (2009). The Tailored Activity Program to reduce behavioral symptoms in individuals with dementia: Feasibility, acceptability, and replication potential. The Gerontologist, 49(3), 428-439. doi:10.1093/geront/gnp087 Grant, J. S., & Davis, L. T. (1997). Selection and use of content experts in instrument development. Research in Nursing & Health, 20, 269-274. Goverover, Y., Kalmar, J., Gaudino-Goering, E., Shawaryn, M., Moore, N. B., Halper, J., & DeLuca, J. (2005). The relation between subjective and objective measures of everyday life activities in persons with multiple sclerosis. Archives of Physical Medicine and Rehabilitation, 86, 2303-2308. Hahn, B., Baum, C., Moore, J., Ehrlich-Jones, Spoeri, S., Doherty, M., & Wolf, T. (2014). Brief Report-Development of additional tasks for the Executive Function Performance Test. American Journal of Occupational Therapy, 68, e241-e246. http://dx.doi.org/10.5014/ajot.2014.008565 Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content Validity in Psychological Assessment: A Functional Approach to Concepts and Methods. Psychological Assessment, 7(3), 238-247. CONTENT-ORIENTED VALIDATION Office of Disease Prevention and Health Promotion. (2014). Dementias, including Alzheimers Disease. Retrieved from https://www.healthypeople.gov/2020/topicsobjectives/topic/dementias-including-alzheimers-disease/objectives Herbert, L. E., Weuve, J., Scherr, P. A., & Evans, D. A. (2013). Alzheimer disease in the United States (2010-2050) estimated using the 2010 census. Neurology. Advance online publication. doi: 10.1212/WNL.0b013e31828726f5Neurology10.1212/WNL.0b013e31828726f5 Henry, A. D., Moore, K., Wuinlivan, M., & Triggs, M. (1998). The relationship of the Allen Cognitive Level Test to demographics, diagnosis, and disposition among psychiatric inpatients. American Journal of Occupational Therapy, 52(8), 638-643. Katz, N. (2006). Routine Task Inventory-Expanded. Retrieved from http://www.allencognitive-network.org/images/stories/pdf_files/rtimanual2006.pdf Katz, N., Tadmore, I., Felzen, B., & Hartman-Maeir, A. (2007). Validity of the Executive Function Performance Test in individuals with schizophrenia. Occupational Therapy Journal of Research, 27, 1-8. Keller, S. & Hayes, R. (1998). The relationship between the Allen Cognitive Level Test and the Life Skills Profile. American Journal of Occupational Therapy, 52(10), 851-856. doi: 10.5014/ajot.52.10.851 Kibayashi, K., Sumida, T., Shojo, H., & Hanada, M. (2007). Dementing diseases among elderly persons who suffered fatal accidents: A forensic autopsy study. American Journal of Forensic Medicine and Pathology, 28(1), 7379. 47 CONTENT-ORIENTED VALIDATION 48 Lehmann, S. W., Black, B. S., Shore, A., Kasper, J., & Rabins, P. V. (2010). Living alone with dementia: Lack of awareness adds to functional and cognitive vulnerabilities. International Psychogeriatrics, 22(5), 778784. doi: 10.1017/S1041610209991529 Lee, C. C., Czaja, S. J., & Schulz, R. (2010). The moderating influence of demographic characteristics, social support, and religious coping on the effectiveness of a multicomponent psychosocial caregiver education in three radical ethnic groups. Journal of Gerontology: Psychological Sciences, 65B(2),185-192. doi: 10.1093/geronb/ghp131 Lee, S. N., Gargiullo, A., Brayman, S., Kinsey, J. C., Jones, H. C., & Shotwell, M. (2003). Adolescent performance on the Allen Cognitive Levels Screen. American Journal of Occupational Therapy, 57(3), 342-346. doi: 10.5014/ajot.57.3.342 Levy, L. L. & Burns, T. (2011). The cognitive disabilities model in 2011. In N. Katz (Ed.), Cognition, occupation, and participation across the life span: Neuroscience, neurorehabilitation, and models of intervention in occupational therapy 3 ed. (pp. 407rd 441). Bethesda, MD: AOTA Press. Lynn, M. (1986). Determination and quantifications of content validity. Nursing Research, 35(6), 382-385. doi:10.1097/00006199-1986-11000-00017 Malmgreen, C., Graham, P. L., Shortridge-Baggett, L. M., Courtney, M., & Walsh, A. (2009). Establishing content validity of a survey research instrument. Journal of nurses in staff development, 25(6), E14-E18. CONTENT-ORIENTED VALIDATION 49 McAnanama, E. P., Rosgosin-Rose, M. L., Scott, E. A., Joffe, R. T., & Kelner, M. (1999). Discharge planning in mental health: The relevance of cognition to community living. American Journal of Occupational Therapy, 53(2), 129-137. McCraith, D. B, Austin, S. L. & Earhart, C. A. (2011). The cognitive disabilities model in 2011. In N. Katz (Ed.), Cognition, occupation, and participation across the life span: Neuroscience, neurorehabilitation, and models of intervention in occupational therapy 3 ed. (pp. 383-406). Bethesda, MD: AOTA Press. rd McKhann, G. M., Knopman, D. S., Chertkow, H., Hyman, B. T., Jack, C. R., Kawas, C. H., . . . Phelos, C. H. (2011). The diagnosis of dementia due to Alzheimers disease: Recommendations from the National Institute on Aging-Alzheimers Association workgroups on diagnostic guidelines for Alzheimers disease. The Journal of the Alzheimers Association, 7(3), 263-269. Retrieved from https://doi.org/10.1016/j.jalz.2011.03.005 Miranda-Castillo, C., Woods, B., & Orrell, M. (2010). People with dementia living alone: What are their needs and what kind of support are they receiving? International Psychogeriatrics, 22(4), 60717. doi: 10.1017/S104161021000013X Abstract retrieved from https://www.ncbi.nlm.nih.gov/pubmed/20214844 Nourhashemi, F., Amouyal-Barkate, K., Gillette-Guyonnnet, S., Cantet, C., & Vellas, B. (2005). Living alone with Alzheimers disease: Cross-sectional and longitudinal analysis in the Real.FR study. The Journal of Nutrition, Health & Aging, 9, 117120. CONTENT-ORIENTED VALIDATION 50 Penny, N. H, Mueser, K. T., & North, T. C. (1995). The Allen Cognitive Level Test and social competence in adult psychiatric patients. American Journal of Occupational Therapy, 49(5), 420-427. doi:10.5014/ajot.49.5.420 Pinquart, M. & Srensen, S. (2003). Association of stressors and uplifts of caregiving with caregiver burden and depressive mood: A meta-analysis. The Journals of Gerontology: Series B, 58(2), 112-128. Polit, D. F. & Beck, C. T. (2006). The Content validity index: Are you sure you know whats being reported? Critique and recommendations. Research in Nursing and Health, 29, 489-497. doi:10.1002/nur.20147 Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30, 459- 467. doi:10.1002/nur.20199 Raweh, D. V. & Katz, N. (1999). Treatment effectiveness of Allens cognitive disabilities model with adult schizophrenic outpatients: A pilot study. Occupational Therapy in Mental Health, 14(4), 65-77. Reisberg, B., Franssen, E. H., Souren, Liduin. E. M., Auer, S. R., Akram, I., & Kenowsky, S. (2002). Evidence and mechanisms of retrogenesis in Alzheimers and other dementias: Management and treatment import. American Journal of Azheimers Disease & Other Dementias, 17(4), 202-212. Renda, M., Lee, S., Keglovits, M., & Somerville, E. (2016). The role of occupational therapy in reducing hospital readmissions. OT Practice, 21(5). CE-1-CE-8. CONTENT-ORIENTED VALIDATION Rogers, J. C., Holm, M. B., Burgio, L. D., Hsu, C., Hardin, J. M., & McDowell, B. J. (2000). Excess disability during morning care in nursing home residents with dementia. International Psychogeriatrics, 12(2), 267-282 Rowe, M. A., Ahn, H., Benito, A. P., Stone, H., Wilson, A., & Kairalla, J. (2010). Injuries and unattended home exits in persons with dementia: a 12-month prospective study. American Journal of Alzheimers Disease and Other Dementias, 25(1), 27-31. doi: doi:10.1177/1533317508323138 Rubenstein, L. Z. & Josephson, K. R. (2006). Falls and their prevention in elderly people: What does the evidence show? Medical Clinics of North America, 90(5), 80724. Rubio, D. M., Berg-Weger, M., Tebb, S. S. Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94-104. Schaber, P., Stallings, E., Brogan, C., & Ali, F. (2016). Interrater reliability of the Revised Cognitive Performance Test (CPT): Assessing cognition in people with neurocognitive disorders. American Journal of Occupational Therapy, 70, 7005290010p1. doi: 10.5014/ajot.2016.019166 Sireci, S. G. & Sukin, T. (2014). Test Validity. In K. F. Geisinger (Editor-in-Chief). APA Handbook of Testing and Assessment in Psychology: Volume 1 Test Theory and Testing and Assessment in Industrial and Organizational Psychology. (p. 79). Washington, DC: American Psychological Association. Slaughter, S. E., Eliasziw, M., Morgan, D., & Drummond, N. (2011). Incidence and predictors of excess disability in walking among nursing home residents with middle-stage 51 CONTENT-ORIENTED VALIDATION 52 dementia: a prospective cohort study, International Psychogeriatrics. 23(1), 54-64. doi:10.1017/S1355617711001238 Slaughter, S. E. & Hayduck, L. A. (2012). Contributions of environment, comorbidity, and stage of dementia, to the onset of walking and eating disability in long-term care residents, Journal of the American Geriatric Society, 60, 1624-1631. doi: 10.1111/j.1532-5415.2012.04116.x Srensen, S., Duberstein, P., Gill, D., & Pinquart, M. (2006). Dementia care: Mental health effects, intervention strategies, and clinical implications. The Lancet Neurology, 5(11), 961-973. Steinberg, M., Sheppard, J. M., Tschanz, J. T., Norton, M. C., Steffens, D. C., Breitner, J. C., & Lyketsos, C. G. (2003). The incidence of mental and behavioral disturbances in dementia: The Cache County Study. Journal of Neuropsychiatry and Clinical Neurosciences, 15(3), 340345. Tierney, M. C., Charles, J., Naglie, G., Jaglal, S., Kiss, A., & Fisher, R. H. (2004). Risk factors for harm in cognitively impaired seniors who live alone: A prospective study. Journal of the American Geriatrics Society, 52(9), 14351441. Tierney, M. C., Snow, W. G., Charles, J., Moineddin, R., & Kiss, A. (2007). Neuropsychological predictors of self-neglect in cognitively impaired older people who live alone. American Journal of Geriatric Psychiatry, 15(2), 1408. United States Department of Health & Human Services. (2009). Code of Federal Regulations. Retrieved from https://www.hhs.gov/ohrp/regulations-andpolicy/regulations/45-cfr-46/#46.102 CONTENT-ORIENTED VALIDATION 53 Velligan, D.I., Bow-Thomas, C.C., Mahurin, R., Miller, A., Dassori, A., and Erdely, F. (1998). Concurrent and predictive validity of the Allen Cognitive Levels Assessment. Psychiatry Research, 80(3), 287-298. Velligan, D. I., True, J. E., Lefton, R. S., Moore, T. C., & Flores, C. V. (1995). Validity of the Allen Cognitive Levels Assessment: A tri-ethnic comparison. Psychiatry Research, 56(2), 101-109. Walstra, G. J., Teunisse, S., van Gool, W. A., & van Crevel, H. (1997). Symptomatic treatment of elderly patients with early Alzheimers disease at a memory clinic. Journal of Geriatric Psychiatry and Neurology, 10(1), 33-38. Warchol, K. (2004). An interdisciplinary dementia program model for long-term care. Topics in geriatric rehabilitation, 20(1), 59-71. Warchol, K. (2006). Facilitating functional and quality-of-life potential: Strengthbased assessment and treatment for all stages of dementia. Topics in Geriatric Rehabilitation, 22(3), 213-227. Warchol, K., Copeland, C., & Ebell, C. (2006). Dementia Therapy: Achieving Positive Outcomes for the Personwith Dementia Self-Study Manual. Chesterfield, MO: Dementia Care Specialists, Inc. World Health Organization. (2002). Towards a Common Language for Functioning, Disability and Health ICF. Geneva, Switzerland. Retrieved from http://www.who.int/classifications/icf/icfbeginnersguide.pdf?ua=1 Yaffe, K., Fox, P, Newcomer, R. Sands, L., Lindquist, K., Dane, K., & Covinsky, K. E. (2002). Patient and caregiver characteristics and nursing home placements in patients CONTENT-ORIENTED VALIDATION 54 with dementia. Journal of the American Medical Association, 287(16), 2090-2097. doi:10.1001/jama.287.16.2090 Yuen, H. K., & Austin, S. L. (2014). Centennial VisionSystematic review of studies on measurement properties of instruments for adults published in the American Journal of Occupational Therapy, 2009-2013. American Journal of Occupational Therapy, 68, e97-e106. http://dx.doi.org/10.5014/ajot.2014.011171 Ziv, N., Roitman, D. M., & Katz, N. (1999). Problem solving, sense of coherence and instrumental ADL of elderly people with depression and normal control group. Occupational Therapy International, 6(4), 243-250. CONTENT-ORIENTED VALIDATION 55 Appendix A CONTENT-ORIENTED VALIDATION 56 Appendix B CONTENT-ORIENTED VALIDATION 57 Appendix C ...