Search

... INCREASING UTILIZATION OF OCCUPATIONAL THERAPY AND PHYSICAL THERAPY SERVICES TO PROVIDE MOBILITY THERAPIES IN THE PICU: A QUALITY IMPROVEMENT PROJECT Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Michelle A. Kiger, MHS, OTR/L Copyright November 25, 2019 By: Michelle A. Kiger, MHS, OTR/L All rights reserved Approved by: Kathy Martin, PT, DHSc Committee Chair Mark V. Paterno, PT, PhD, MBA, SCS, AT-Ret Committee Member Carolyn P. Luzader, MS Committee Member Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis INCREASING UTILIZATION OF OT AND PT IN THE PICU Increasing Utilization of Occupational Therapy and Physical Therapy Services to Provide Mobility Therapies in the PICU: A Quality Improvement Project Michelle A Kiger, MHS, OTR/L University of Indianapolis Cincinnati Childrens Hospital Medical Center 1 INCREASING UTILIZATION OF OT AND PT IN THE PICU 2 Abstract There is a growing body of evidence demonstrating early mobilization for critically ill children is safe and feasible. Occupational therapists and physical therapists have distinct skills and abilities which make them an integral part of an interdisciplinary early mobilization program in the pediatric intensive care unit (PICU). The incorporation of occupational therapy and physical therapy services in an early mobilization program assists in decreasing the negative effects of immobility and inactivity through the facilitation of active movement and participation in age appropriate functional activities. Despite consults being placed within 72 hours of admission in the PICU at Cincinnati Childrens, barriers existed which impacted the ultimate delivery of OT and PT services to patients. The purpose of this study was to assess whether the desired improvement in the delivery of occupational therapy and physical therapy services in the PICU was achieved when interventions were applied using quality improvement methodology. A quality improvement approach, guided by Cincinnati Childrens rapid cycle improvement collaborative roadmap, aimed to identify whether interventions applied during a 120 day quality improvement project were effective in transforming the care delivery processes for therapy services in the PICU at a single institution as measured by the ability to produce a 20% improvement in the percentage of scheduled occupational therapy and physical therapy treatments which were delivered. The PICU population included in this quality improvement project consisted of patients with (1) an occupational therapy and physical therapy consult, (2) a scheduled occupational therapy and/or physical therapy appointment, and (3) a bed on the PICU floor. The study period was from December 18, 2018 through April 16, 2019. Adoption of several interventions increased the median percentage of scheduled occupational therapy and physical therapy treatments which were delivered from 48.6% at baseline to 56%. INCREASING UTILIZATION OF OT AND PT IN THE PICU Acknowledgements I would like to thank my sons, Sam and Wes and my husband, Sam for their unending support during my pursuit of a DHSc and this doctoral project. I am thankful for the continuous support of coworkers who participated in this project. Thank you to my family and friends for their encouragement, support, and assistance during this time. I am eternally grateful to my mother for encouraging me to be a life-long learner. I would like to thank my committee Dr Kathy Martin, Dr Mark Paterno, and Carolyn Luzader for sharing their knowledge, providing encouragement, and continuous support throughout this process. I will forever be grateful for their commitment to assisting me with this project. Lastly, I would like to thank the numerous professors and teachers for the support they have given me through the years. . 3 INCREASING UTILIZATION OF OT AND PT IN THE PICU 4 Table of Contents Title Page ........................................................................................................................................ 1 Abstract ........................................................................................................................................... 2 Acknowledgements ......................................................................................................................... 3 Table of Contents ............................................................................................................................ 4 Chapter 1: Introduction ................................................................................................................... 6 Problem Statement ...................................................................................................................... 8 Purpose Statement ....................................................................................................................... 9 Research Questions ..................................................................................................................... 9 Significance of the Study .......................................................................................................... 10 Definition of Terms ................................................................................................................... 10 Literature Review.......................................................................................................................... 12 Occupational Therapy and Physical Therapy Services in the PICU ......................................... 15 Early Mobilization in the PICU ................................................................................................ 17 Barriers to Early Mobilization in the PICU .............................................................................. 18 Quality Improvement in Healthcare .......................................................................................... 21 Quality Improvement at CCHMC ............................................................................................. 22 Method .......................................................................................................................................... 24 Study Design ............................................................................................................................. 24 Participants ................................................................................................................................ 24 Procedures ................................................................................................................................. 26 Data Collection, Management, and Analysis ............................................................................ 32 Results ........................................................................................................................................... 33 Data ........................................................................................................................................... 33 Statistical Analysis .................................................................................................................... 35 PDSA Cycles ............................................................................................................................. 36 Analyzing the Run Chart/Control Chart ................................................................................... 37 INCREASING UTILIZATION OF OT AND PT IN THE PICU 5 Discussion ..................................................................................................................................... 42 Limitations ................................................................................................................................ 47 Implications for Practice and Future Research ......................................................................... 48 Conclusion .................................................................................................................................... 49 References ..................................................................................................................................... 50 Figures........................................................................................................................................... 60 Appendices .................................................................................................................................... 69 INCREASING UTILIZATION OF OT AND PT IN THE PICU 6 Chapter 1: Introduction Increasing Utilization of Occupational Therapy and Physical Therapy Services to Provide Mobility Therapies in the PICU: A Quality Improvement Initiative In the United States, over 230,000 children were admitted to a pediatric intensive care unit (PICU) in 2001 (Randolph, Gonzales, Cortellini, & Yeh, 2004). The majority (98%) of children with a critical illness admitted to a PICU survive (Pollack, Holubkov, Funai, Clark, Berger, et al., 2014; Pollack, Holubkov, Funai, Clark, Moler, et al., 2014). Evidence indicates there is the potential to develop new morbidities associated with a PICU admission (Choong, Fraser, et al., 2018; Hopkins, Choong, Zebuhr, & Kudchadkar, 2015; Pollack, Holubkov, Funai, Clark, Berger, et al., 2014). Pollack, Holubkov, Funai, Clark, Berger, et al. (2014) found that at discharge from the PICU, the prevalence of new morbidities was 4.8% for children who survive their critical illness, which is twice the mortality rate of 2.0%. New morbidities may occur in mental status, sensory function, communication skills, motor function, feeding, and respiratory function (Pollack, Holubkov, Funai, Clark, Berger, et al., 2014). The highest proportion of new morbidities occurred in respiratory, motor, and feeding domains (Pollack, Holubkov, Funai, Clark, Berger, et al., 2014). Intensive care unit acquired weakness (ICUAW) is one such consequence of critical illness and prolonged immobility and inactivity (Kukreti, Shamim, & Khilnani, 2014). These sequelae have the potential to lead to persistent functional impairments in children who survive a critical illness (Ong, Lee, Leow, & Puthucheary, 2016). Early rehabilitation and mobilization in the PICU is a newer treatment approach which typically includes active mobility interventions provided within 24-72 hours of hospitalization in order to maintain or restore strength and physical activity. (Choong, Canci, et al., 2018; Choong et al., 2015; Cui et al., 2017; Wieczorek et al., 2016; Wieczorek, Burke, Al-Harbi, & INCREASING UTILIZATION OF OT AND PT IN THE PICU 7 Kudchadkar, 2015). Studies on early mobilization (EM) in the pediatric population are more recent and consist of observational studies and 2 randomized control trials, all with limited samples sizes (Abdulsatar, Walker, Timmons, & Choong, 2013; Betters et al., 2017; Choong et al., 2017; Choong et al., 2015; Fink et al., 2019; Tsuboi et al., 2019; Tsuboi et al., 2017; Wieczorek et al., 2016). Despite differences in EM protocols and variability of study populations, all studies found that EM in the PICU is a feasible and safe intervention (Abdulsatar et al., 2013; Betters et al., 2017; Choong et al., 2017; Choong et al., 2015; Cuello-Garcia, Mai, Simpson, Al-Harbi, & Choong, 2018; Fink et al., 2019; Piva, Ferrari, & Schaan, 2019; Tsuboi et al., 2019; Tsuboi et al., 2017; Wieczorek et al., 2016; Wieczorek et al., 2015). Active mobility interventions are intended to help improve outcomes through mitigation of new morbidities associated with a PICU admission and maximization of functional mobility (Choong, Canci, et al., 2018). The primary objectives of EM are to reduce PICU and hospital length of stay and optimize functional recovery (Choong, Fraser, et al., 2018). Therefore, once a patient is stabilized, focus on early rehabilitation and mobilization may aid in maximizing functional outcomes (Choong, Canci, et al., 2018; Turner et al., 2011). Although participation in EM is feasible and safe, readiness for participation in EM ought to take into consideration both contraindications and precautions to mobilization for children with a critical illness (Choong, Canci, et al., 2018). Children who are critically ill are a heterogeneous population with unique needs based upon their age, maturation, cognitive abilities, and baseline function. EM of children in the PICU should be developmentally appropriate, individualized, and serve to decrease the negative effects of immobility and inactivity such as muscle weakness and deconditioning. (Choong, Canci, et al., 2018). EM activities should be progressed in a step-wise fashion aimed at achieving functional mobility and INCREASING UTILIZATION OF OT AND PT IN THE PICU 8 increasing muscle strength (Choong, Canci, et al., 2018). Initiating rehabilitation services early may support a childs ability to maximize functional mobility during and after a PICU admission (Choong et al., 2017). The ability to deliver occupational therapy (OT) and physical therapy (PT) services in the PICU may be impacted by patient-related barriers, structural barriers, PICU culture barriers, and process-related barriers. A survey of PICU healthcare providers found that 73% believed the amount of rehabilitation services provided by OT and PT to children who were critically ill was inadequate (Joyce et al., 2018). Despite a critically ill childs need for rehabilitation services, past studies have noted limitations in the availability of therapy resources to support those needs in some settings (Choong, Fraser, et al., 2018; Cremer, Leclerc, Lacroix, & Ploin, 2009; Cui et al., 2017; Zheng et al., 2018). Additionally, two studies reported that therapists experienced frequent deferral of therapy sessions by nursing, impacting their ability to deliver care (Cui et al., 2017; Fink et al., 2019). Barriers to a therapist delivering care limits their ability to positively impact functional mobility outcomes. Problem Statement In May 2017, a multidisciplinary team engaged in a quality improvement (QI) project to decrease the negative effects of immobility and inactivity for critically ill patients in the PICU at Cincinnati Childrens Hospital (CCHMC). The team followed the QI methodology developed at CCHMC. This resulted in the development and implementation of interventions which were successful, reliable, and sustainable to carrying out an EM program for patients with and without mechanical ventilation. CCHMC adapted the PICU UP! Guidelines (Wieczorek et al., 2016) to the local context and implemented this program for patients who would benefit from EM. Increasing the opportunity for OT and PT to deliver services to patients in the PICU was a key INCREASING UTILIZATION OF OT AND PT IN THE PICU 9 strategy and driver to the implementation of the EM program at CCHMC. During the first 90-day QI project the team identified that very few patients were receiving an OT/PT consult soon after admission to the PICU and recognized this as the problem their QI project would address. The QI project resulted in an increase in OT/PT consults being requested by physicians and nurse practitioners within 72 hours of admission from 8% to 22%. Despite improvements in consults being placed within 72 hours of admission in the PICU, barriers existed which impacted the ultimate delivery of OT and PT services to patients. Beginning in October 2018, a subset of the multidisciplinary team from the first QI project participated in a second QI project. The QI project followed the Rapid Cycle Improvement Collaborative (RCIC) roadmap developed at CCHMC (Kaminski, Schoettker, Alessandrini, Luzader, & Kotagal, 2014) in order to affect an increase in the percentage of OT and PT sessions delivered to patients in the PICU who had a consult for such services. Applying this QI methodology, the team set out to identify and implement successful, reliable, and sustainable interventions with the goal to support increased OT and PT care delivery to patients in the PICU. Purpose Statement The purpose of this study was to assess whether the desired improvement in the delivery of OT and PT services in the PICU was achieved when interventions were applied using QI methodology. Research Questions The primary research aim was to identify if interventions applied during a 120 day QI project were effective in transforming the care delivery processes for therapy services in the PICU as measured by the ability to produce a 20% improvement in the percentage of scheduled INCREASING UTILIZATION OF OT AND PT IN THE PICU 10 OT and PT treatments which were delivered. A secondary research question was to report the percentage of scheduled OT and PT appointments in the PICU which were deferred and the reasons consults were deferred in order to prioritize interventions on the problem which have the greatest potential to impact the delivery of OT and PT treatments. Significance of the Study Occupational therapists and physical therapists have distinct skills and abilities which make them an integral part of an interdisciplinary EM program in the PICU. The objective of EM programs is to decrease the negative effects of immobility and inactivity through the facilitation of active movement and participation in age appropriate functional activities. Minimizing the negative effects of immobility and inactivity and improving a patients functional outcomes after hospitalization are beneficial to the patient, family, and society at large. The incorporation of OT and PT services in an EM program will support these objectives. This study reports the QI process followed to improve the consistent delivery of OT and PT services in the PICU and could have generalizability in other, similar settings and patient populations. Definition of Terms Early mobilization typically includes active mobility interventions intended to maintain or restore strength, physical activity, and provided within 24-72 hours of hospitalization. (Choong, Canci, et al., 2018; Choong et al., 2015; Cui et al., 2017; Wieczorek et al., 2016; Wieczorek et al., 2015). Function involves all body functions (physical, cognitive, and psychosocial), activities of daily living, and participation in age appropriate life activities in the home, at school, and in the community (World Health Organization, 2007). INCREASING UTILIZATION OF OT AND PT IN THE PICU 11 Intensive care unit acquired weakness (ICUAW) is characterized by wide spread musculoskeletal weakness often due to critical illness polyneuropathy or critical illness myopathy (Kukreti et al., 2014). Acquired morbidity is a condition or disease that is the result of a complication or undesirable side effect following medical treatment. (Choong et al., 2015; Joyce et al., 2018)) Rehabilitation includes treatment or treatments intended to facilitate the process of recovery from injury, illness, or disease (Choong et al., 2014, p. e271). Session deferral occurs when an occupational therapist or physical therapist attempts to provide services to a patient and the services are put off, delayed, or postponed. Morbidity refers to the impairment of functional health status (Odetola, 2014) Chest PT are "Physical methods to improve ventilation, ventilation/perfusion (V/Q) matching, breathing mechanics, and airway secretions clearance (e.g., percussion techniques, manual facilitation of chest wall movement, and deep breathing exercises [including blowing bubbles and incentive spirometry])" (Choong et al., 2013) Passive range of motion (PROM) includes passive repositioning of patient or passive stretching of their limbs and joints. Passive means the patient does not voluntarily participate in the activity" (Choong et al., 2013) Active range of motion (AROM) may include exercises and stretches that are taught to a patient to do independently and includes neurodevelopmental play"; active infers patient participation, no matter how little (Choong et al., 2013) Mobility device refers to "activities done with a device that facilitates limb movement (i.e., cycle ergometer)" (Choong et al., 2013) INCREASING UTILIZATION OF OT AND PT IN THE PICU 12 Bed mobility refers to "activities done while patient is recumbentbut requires active participation of the patient; for example, active or active-assisted repositioning in bed; rolling from side to side; and bridging (i.e., pelvic or hip lifts)" (Choong et al., 2013) Transfers refers to patient transferring from one surface to the other (e.g., from bed to chair/commode, sitting or dangling on edge of bed)" (Choong et al., 2013). These activities may be active or passive, may occur with varying degrees of caregiver assistance and supervision, or may be performed independently (Choong, Canci, et al., 2018). Pre-gait activities refer to "assisting the patient in exercises prior to ambulation" (Choong et al., 2013). Ambulation refers to "gait training of the patient, with or without assistance by therapist or device (e.g., walker)" (Choong et al., 2013). Huddles refers to a brief (i.e. 15-20 minute) meeting to share and discuss important information necessary to keep the work on track. Reliability in healthcare is the capability of a process, procedure or health service to perform its intended function in the required time under existing conditions (Nolan, Resar, Griffin, & Gordon, 2004). Literature Review A PICU is the unit in a hospital that delivers the highest level of medical care to children with severe illness (Odetola, 2014). Patients admitted to the PICU often include those with severe respiratory illnesses, serious infections, cardiopulmonary conditions, fluid and electrolyte imbalances, require post-surgical care, or those involved in a serious trauma (Edwards et al., 2012; Krmpotic & Lobos, 2013). The medical staff in the PICU provide intensive therapies that may not be available on other medical units within a hospital (Odetola, Clark, Freed, Bratton, & INCREASING UTILIZATION OF OT AND PT IN THE PICU 13 Davis, 2005; Randolph et al., 2004). Some of these more intensive therapies include mechanical ventilation and certain medicines that can be given only under close medical supervision (Odetola et al., 2005; Randolph et al., 2004). As such, the focus of care in the PICU is complex management of some of the highest risk patients. Patients admitted to the PICU often have complex chronic conditions (CCC). Childhoodonset chronic conditions, including congenital heart abnormalities, cerebral palsy, and chromosomal abnormities, have been reported in 53% of children admitted to PICUs in the United States (Edwards et al., 2012), 67% in France (Cremer et al., 2009), and 70% in Canada (Choong et al., 2017). Krmpotic and Lobos (2013) reported the most common preexisting medical conditions for children with an unplanned admission to the PICU were respiratory disease (26%) and neurologic/ neuromuscular (15%) conditions. Children with CCC represent a large percentage of the patient population admitted to the PICU, as indicated by multi-site and single-site studies (Choong et al., 2017; Cremer et al., 2009; Edwards et al., 2012; Krmpotic & Lobos, 2013). The rate of PICU growth over the past 2 decades is significant. In 1995, there were 306 PICUs in the United States (Randolph et al., 2004). By 2004, that number had grown to 337 hospitals with approximately 4,044 beds admitting 58 patients per PICU bed annually (Odetola et al., 2005). The average number of beds in a PICU in 2014 was 12 (interquartile range: 817 beds) (Odetola et al., 2005). The majority of intensive care units (ICUs) are either pediatric or adult and few mix care between these populations. In one survey, less than 6% of ICUs in the United States provided care to both adults and children suffering from a critical illness in the same area (Odetola et al., 2005). At CCHMC, one of the nations larger ICUs caring for patients INCREASING UTILIZATION OF OT AND PT IN THE PICU 14 021 years of age, more than 2,000 critically ill infants and children are admitted to the 35-bed PICU annually (Cincinnati Children's Hospital Medical Center, 2018). Mortality has declined in areas where PICU services are accessible as a result of advances in intensive therapies and biomedical science (Odetola, 2014). While the majority of children survive a critical illness, studies suggest many are left with functional impairments and decreased quality of life (QOL) (Aspesberro, Mangione-Smith, & Zimmerman, 2015; Choong et al., 2013; Odetola, 2014; Ong et al., 2016). Following critical illness, evidence suggests children are at risk of acquiring new morbidities and experiencing functional deterioration (Choong, Fraser, et al., 2018; Choong et al., 2013; Pollack, Holubkov, Funai, Clark, Berger, et al., 2014; Wieczorek et al., 2015). PICU survivors are at risk of ICUAW, neurocognitive impairments, mental health sequelae, reduced QOL, caregiver burden, and functional impairments (Choong et al., 2014; Hopkins et al., 2015; Joyce et al., 2018). The highest proportion of acquired morbidities occur in respiratory, motor, and feeding domains (Pollack, Holubkov, Funai, Clark, Berger, et al., 2014). These sequelae may lead to persistent functional impairments in pediatric survivors of critical care (Odetola, 2014; Ong et al., 2016). In two single-site studies, newly acquired functional deterioration following critical illness was found in at least 80% of patients admitted to the PICU (Choong et al., 2017; Choong, Canci, et al., 2018). Functional deterioration affected daily activities, mobility, social/cognitive, and responsibility domains, as measured by the Pediatric Evaluation of Disability InventoryComputer Adaptive Test (PEDI-CAT) (Choong et al., 2017; Choong, Fraser, et al., 2018). Six months after discharge, 67.1% of patients demonstrated some functional recovery in one or more domains of function and 24% had not yet returned to baseline mobility function (Choong, Fraser, et al., 2018). Choong et al. (2017) found that mobility seemed to be the slowest to recover. This INCREASING UTILIZATION OF OT AND PT IN THE PICU 15 is important because functional decline has been shown to prolong hospital length of stay (Choong, Fraser, et al., 2018). While functional deterioration is common and can result in disability, functional deterioration and acquired impairments that result in disability are considered an undesired outcome of a PICU admission (Odetola, 2014). Safety concerns and severity of illness often give rise to the perception that patients are too ill to be mobilized, resulting in prolonged periods of bed rest (Choong et al., 2014; Choong et al., 2013). Prolonged immobility is thought to increase the risk of critical illness-acquired morbidities such as ICUAW. ICUAW is characterized by polyneuropathy and myopathy-induced muscle weakness (Kukreti et al., 2014). Kukreti et al. (2014) found that ICUAW in children has a negative impact on morbidity, length of stay, and possibly mortality. Although this is a common ICU acquired morbidity in adults (Fan, 2012), Kukreti et al. (2014) concluded that future research was necessary to understand the prevalence of ICUAW in children. Occupational Therapy and Physical Therapy Services in the PICU In the PICU, occupational therapists and physical therapists treat patients with a variety of ages and conditions. Physical therapists work with children to improve their strength, endurance, and mobility while occupational therapists work with children to facilitate mobilization, maximize function, and integrate age appropriate occupational-based activities, Both OT and PT work to prevent the many complications of prolonged bedrest and immobility. Neuromuscular complications following a critical illness are common. In the absence of established pediatric clinical practice guidelines for these populations, therapists are required to use their clinical reasoning skills to integrate medical information with OT or PT knowledge to provide evidence based care. INCREASING UTILIZATION OF OT AND PT IN THE PICU 16 OT and PT interventions in the PICU have often included both non-mobility and mobility therapies to address these complications. Non-mobility therapies are those which enhance respiratory function (chest PT), passive positioning, splinting, or passive range of motion (PROM) and stretching (Choong, Canci, et al., 2018; Choong et al., 2014; Choong et al., 2013; Wieczorek et al., 2016). Mobility therapies include active range of motion (AROM) or active strengthening exercises, mobilization devices, bed mobility, transfers, pre-gait activities, and ambulation (Choong, Canci, et al., 2018; Choong et al., 2015; Choong et al., 2014; Choong et al., 2013; Wieczorek et al., 2016). The role of OT and PT in the PICU at our institution is trending toward the delivery of more active interventions. Physical therapists have long supported patients in the PICU through non-mobility therapies such as PROM, splinting, passive positioning, and respiratory PT (chest PT). Hawkins and Jones (2015) found that the primary intervention provided by PT in the PICU to mechanically ventilated patients was chest PT. Chest PT is for the management of secretion clearance and has been shown to be safe and effective (Hawkins & Jones, 2015). In addition, chest PT was the most frequent intervention requested by physicians (50.5%) in tertiary care pediatric critical care units across Canada (Choong et al., 2014). Consequently, interventions most often delivered by physiotherapists in the Pediatric Critical Care Unit (PCCU) in Canada were chest PT and PROM (77.8%) (Choong et al., 2013). In a survey of Canadian practice with critically ill children, 70.4% of physiotherapists provided pre-gait and ambulation interventions sometimes or infrequently (Choong et al., 2013). Cui et al. (2017) reported that in 2011 only a third of children admitted to a single-site tertiary care PICU in the United States for 3 days participate in active mobility interventions. Mobility therapies include activities to maximize physical function and muscle strength. These may INCREASING UTILIZATION OF OT AND PT IN THE PICU 17 include both in-bed and out-of-bed activities (Wieczorek et al., 2016). Active mobility interventions are individualized to each patient and progress from in-bed activities to ambulating with or without an assistive device (Choong et al., 2014; Cui et al., 2017; Owens & Tapley, 2018; Wieczorek et al., 2016). Early Mobilization in the PICU EM in the PICU is a fairly new treatment option for the prevention and treatment of neuromuscular complications, such as ICUAW, following a critical illness (Choong, Canci, et al., 2018; Cuello-Garcia et al., 2018; Piva et al., 2019). The definition of EM in children who are critically ill typically includes developmentally age appropriate active mobility interventions intended to maintain or restore strength and function (Choong, Canci, et al., 2018; Cui et al., 2017; Parisien et al., 2016; Wieczorek et al., 2016) initiated within 24-72 hours of admission (Choong et al., 2015; Wieczorek et al., 2016). Although there is a paucity of research related to efficacy outcomes of EM in pediatrics (Choong, Canci, et al., 2018; Cuello-Garcia et al., 2018; Piva et al., 2019; Tsuboi et al., 2019), there is evidence that suggests prolonged immobility should be avoided in children (Kukreti et al., 2014). Growing evidence demonstrates the benefits of EM in critically ill adults. In the adult population, EM has shown to reduce ICU and hospital length of stay, reduce length of time on the ventilator, and improve self-perception of function, and decrease sedation utilization and delirium (Adler & Malone, 2012; Cameron et al., 2015; Needham et al., 2010; Schweickert et al., 2009). Based upon encouraging adult findings, pediatric studies to date have demonstrated that implementation of early mobility activities is safe and feasible (Abdulsatar et al., 2013; Betters et al., 2017; Cameron et al., 2015; Choong et al., 2017; Choong et al., 2015; Fink et al., 2019; Hollander et al., 2014; Piva et al., 2019; Tsuboi et al., 2019; Turner et al., 2011; Wieczorek et al., INCREASING UTILIZATION OF OT AND PT IN THE PICU 18 2016; Wieczorek et al., 2015). A recent retrospective before-after study of pediatric patients who underwent liver transplantation found that patients who received early mobility interventions were able to walk 5 days earlier and had shorter hospital lengths of stay compared to the preearly mobilization period (Tsuboi et al., 2019). Although Tsuboi et al. (2019) findings will aid in understanding the impact of EM on liver transplantation patients, the impact of early mobilization on the efficacy outcomes in critically ill children [as a whole] remains to be seen owing to the paucity of prospective trials and, therefore, the low certainty in the evidence to date (Cuello-Garcia et al., 2018, p. 32). Early pediatric studies have investigated the implementation of EM programs which sets the foundation for future investigation of the efficacy of EM on functional outcomes, hospital length of stay or ICU length of stay in a younger population. Recent pediatric studies have shown benefits of EM implementation including improved time to referral for OT and PT by PICU day 3 (Tsuboi et al., 2019; Wieczorek et al., 2016) and increases in EM activities (Abdulsatar et al., 2013; Choong et al., 2014; Choong et al., 2013; Hollander et al., 2014; Wieczorek et al., 2016). A survey assessing beliefs of PICU providers in regards to EM revealed that all respondents believed EM would be beneficial to patients, and 93% expressed interest in the implementation of EM (Joyce et al., 2018). Despite this interest, implementation of EM in the pediatric setting has been limited due to the heterogeneous nature of this patient population (Wieczorek et al., 2015) and lack of EM practice guidelines specific to pediatric populations (Choong, Canci, et al., 2018). Barriers to Early Mobilization in the PICU Infrastructure and process level barriers to EM in the pediatric PICU have been identified as a lack of protocols, time constraints, competing care priorities, lack of staffing, lack of INCREASING UTILIZATION OF OT AND PT IN THE PICU 19 training and education, lack of space, conflicting view of suitable patients, therapists requirement for a physicians order prior to evaluation, lack of champions to promote EM, pediatric specific equipment, and safety concerns (Choong, Fraser, et al., 2018; Choong et al., 2013; Cui et al., 2017; Hopkins et al., 2015). Patient level barriers include medical instability, concern for device integrity or dislodgement, the presence of an endotracheal tube, and patient or parent refusal (Choong et al., 2017; Choong et al., 2013). The heterogeneity of the pediatric population, particularly with respect to age, cognitive maturity, and sedation needs is a barrier, which further complicates the implementation of EM (Joyce et al., 2018). Barriers specific to the delivery of EM interventions by OT and PT also include structural barriers, patient-related barriers, PICU culture barriers, and process-related barriers. It has been suggested that structural barriers include knowledge gaps in this area by physicians who make referrals for OT and PT services (Choong et al., 2013), lack of clear practice guidelines (Choong et al., 2013; Wieczorek et al., 2016; Wieczorek et al., 2015), time constraints (Choong et al., 2013; Joyce et al., 2018), and excessive workload (Joyce et al., 2018; Zheng et al., 2018). Patient related barriers include parental stress (Choong et al., 2017; Colwell et al., 2019; Parisien et al., 2016) and safety concerns (Choong et al., 2013; Parisien et al., 2016; Wieczorek et al., 2015; Zheng et al., 2018). Cultural barriers impacting the ability of OT and PT to deliver EM interventions to appropriate patients include beliefs that patients are too sick to participate in therapy (Choong et al., 2014; Zheng et al., 2018). In addition, while it is encouraging that children with known baseline motor impairments were five times more likely to receive EM during periods of critical illness, there is also an opportunity to increase awareness of the rehabilitation needs of children with previously normal baseline function who become critically ill (Miura, Wieczorek, Lenker, & Kudchadkar, 2018). INCREASING UTILIZATION OF OT AND PT IN THE PICU 20 Wieczorek et al. (2016) reported that OT and PT services support early rehabilitation for PICU patients and are felt to be a valuable personnel resource by nursing. In spite of this perceived value, deferment of OT and PT sessions is common in the PICU and was been reported to be 21.5% in one tertiary-level PICU (Cui et al., 2017), 25.8% in a single center, pilot, randomized controlled trial (Choong et al., 2017) and 19% for OT and 21% for PT in a randomized control trial of pediatric neurocritical care patients (Fink et al., 2019). The most common reason for deferral of OT and PT sessions was at the request of nursing (Choong et al., 2013; Cui et al., 2017). Reasons for nursing deferral included the necessity for nursing interventions or other diagnostics/procedure at the same time, difficulty calming a child after sedation interruption, sleeping patient, or at the familys request (Cui et al., 2017). Other reasons for deferral of OT and PT sessions included a sleeping patient (Cui et al., 2017), patient not in hospital room (Cui et al., 2017; Fink et al., 2019), patients condition (Cui et al., 2017; Fink et al., 2019; Wieczorek et al., 2016), patient or family refusal (Fink et al., 2019; Wieczorek et al., 2016), and unavailable equipment (Wieczorek et al., 2016). Deferral of services may have contributed to 75% of PICU clinicians indicating they believe patients did not receive adequate OT and PT services (Joyce et al., 2018). Successful implementation EM has taken into consideration many of these barriers. Mitigation strategies to address conflicting views of suitable patients have included parameters for inclusion in EM (Van Damme, Flori, & Owens, 2018; Wieczorek et al., 2016) and clinical recommendations which identify contraindications and precautions (Betters et al., 2017; Choong, Canci, et al., 2018; Tsuboi et al., 2019; Van Damme et al., 2018). Safety concerns have been addressed through a premobilization safety checklist (Betters et al., 2017; Wieczorek et al., 2016) and criteria for pausing EM activities and reassessing the patient before continuing has INCREASING UTILIZATION OF OT AND PT IN THE PICU 21 been created (Tsuboi et al., 2019; Van Damme et al., 2018; Wieczorek et al., 2016). Parents reported a decrease in anxiety following active participation in their childs care making them less likely to refuse care (Parisien et al., 2016). The identified lack of pediatric EM protocols has resulted in the creation of several institutional protocols (Betters et al., 2017; Van Damme et al., 2018; Wieczorek et al., 2016) as well as the creation of a practice recommendation for EM in children who are critically ill developed by a multidisciplinary panel of experts, using the Institute of Medicine framework (Choong, Canci, et al., 2018). The recent protocols and practice recommendation propose mobilizing in a step wise fashion, based on clinical variables and exclusion criteria, applying individualized developmentally appropriate goals which take into consideration the level of assistance required (Betters et al., 2017; Choong, Canci, et al., 2018; Van Damme et al., 2018; Wieczorek et al., 2016). EM activities may be provided in-bed or out of bed and often progress from in bed movement to sitting edge of bed, sit to stand, standing, transferring to a chair, and the highest level anticipated is ambulation and play (Betters et al., 2017; Cui et al., 2017; Wieczorek et al., 2016). Following implementation of an EM protocol in the PICU, Betters et al. (2017) found that 32% more staff believed mobilizing intubated patients was safe. Consideration of identified barriers and thoughtful planning of mitigation strategies has been shown to lead to successful implementation of EM for critically ill children. Quality Improvement in Healthcare QI is the systematic, data-guided activities designed to bring about immediate, positive changes in the delivery of health care (Baily, Bottrell, Lynn, Jennings, & Hastings, 2006, p. S5). Healthcare organizations may look to QI methodologies to accelerate improvements in processes of care and patient outcomes while decreasing variation in the care delivered to children and their families in collaboration with multiple institutions (Billett et al., 2013; Clauss et al., 2015; INCREASING UTILIZATION OF OT AND PT IN THE PICU 22 Schouten, Hulscher, van Everdingen, Huijsman, & Grol, 2008). There are several QI methodologies developed which have been used in healthcare. Most QI methodologies used in healthcare define a goal, acquire and analyze data used to guide process changes, and they review the results to further refine future changes (Lee & Larson, 2014). A methodology commonly used in healthcare is the Model for Improvement (Langley et al., 2009) which is supported by the Institute for Healthcare Improvement. QI initiatives are a strategy used by many healthcare organizations, including CCHMC, to improve processes of care and patient outcomes. Quality Improvement at CCHMC CCHMC QI foundations are based in a Conceptual Framework using Demings System of Profound Knowledge and Core Methodology using The Model for Improvement (Kaminski et al., 2014). Demings system of profound knowledge is composed of interrelated components, namely appreciation of a system, theory of knowledge, psychology, and understanding variation, which together reinforce one another to accomplish the aim of the system (Deming, 2000). Transformation of systems occurs when improvement is made in all four components of profound knowledge (Deming, 2000), therefore, the components cannot be separated. Deming (2000) believed combining subject matter knowledge (professional knowledge) and profound knowledge (interaction of variation, knowledge, psychology, systems) in novel ways would lead to effective changes for improvement. The Improvement Model asks three basic questions as the framework for improvement: What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? (Langley et al., 2009) INCREASING UTILIZATION OF OT AND PT IN THE PICU 23 The comprehensive improvement capability model at CCHMC includes didactic and hands on learning through action oriented QI projects designed to achieve measurable improvements (Kaminski et al., 2014). There are several programs offered at CCHMC which begin with online modules to acquire knowledge about QI and progress from basic to advance programs designed to acquire broader QI skills which are applied to actionable projects. RCIC is a collaborative in CCHMCs QI educational offerings available to frontline staff to work in teams to deepen their knowledge and skills in improvement science methodology by applying QI methodology to a focused, narrow-scoped question or problem to achieve measurable improvement in 120 days (Kaminski et al., 2014). The RCIC roadmap translates the QI methodology into a sequence of activities and tools for teams as they apply QI methodology to a problem (Luzader, 2018). Use of the RCIC roadmap will allow team members to succinctly identify their current problem, document the current process, identify and analyze process failures, identify root causes, develop a SMART aim and measures, identify key drivers, identify potential interventions, design and execute plan-do-study-act (PDSA) cycles, scale-up successful interventions, and lastly plan for sustainability and spread (Luzader, 2018). Teams have successfully demonstrated improvements in processes of care and patient outcomes through the use of CCHMCs QI methodology. Improvements have been reported for a variety of patient populations, including early identification and management of hospitalized pediatric patients who are at risk for medical deterioration (Parker et al., 2017), a reduction in time to antibiotic delivery in febrile immunocompromised children (Dandoy et al., 2016), and improved systems for identifying patients at risk for acute compartment syndrome (Schaffzin et al., 2013). Improved patient population outcomes following the application of QI methodology, such as for infants with congenital muscular torticollis, have also been reported (Strenk, Kiger, INCREASING UTILIZATION OF OT AND PT IN THE PICU 24 Hawke, Mischnick, & Quatman-Yates, 2017). In addition, QI projects guided by the RCIC roadmap (Figure 1) to accelerate improvements in processes of care while decreasing variation have successfully achieved the desired outcome set forth in the SMART Aim statement (Boyd, 2016). In light of the importance of OT and PT in the delivery of EM interventions to children, and the success of QI methodology in improving processes of care, using the RCIC roadmap is appropriate and meaningful. Method Study Design The objective of this QI project was to apply QI methodology to achieve measurable improvement in a focused, narrow-scoped project in 120 days (Kaminski et al., 2014). The primary research aim was to identify whether interventions applied during a 120 day QI project were effective in transforming the care delivery processes for therapy services in the PICU as measured by the ability to produce a 20% improvement in the percentage of scheduled OT and PT treatments which were delivered. The Institutional Review Boards from CCHMC and from the University of Indianapolis were consulted and apprised of this project. This project did not meet the definition of human subjects research set forth in 45 CFR 46.102(f) since this project was designed and intended to apply improvement methodology to healthcare processes that would benefit the participants of the project. Therefore, informed consent was waived provided information collected did not allow for the identification of participants through the QI project. Participants The PICU at CCHMC is a multidisciplinary inpatient unit that serves critically ill children. The multidisciplinary practice includes physicians (MD/DO), nurse practitioners (APN), registered nurses (RN), patient care attendant (PCA), respiratory therapists (RT), INCREASING UTILIZATION OF OT AND PT IN THE PICU 25 occupational therapist, physical therapist, speech and language pathologists (SLP), dietician, child life specialist, and pharmacists. At CCHMC, there is one primary occupational therapist and one physical therapist who serve this unit along with support from other inpatient OT and PT staff members. All inpatient therapists at CCHMC have acute care training in order to serve patients in the PICU when needed, such as for extended time off or illness of the primary therapists or unexpected increases in census. The inpatient OT and PT services are supported by one program assistant and three unlicensed personnel (rehab tech). This support staff is responsible for scheduling patient appointments on a therapists electronic schedule per therapists instructions which are communicated electronically. Therapists are responsible to manage their own caseload and confirm availability of patients on a daily basis through chart review and care coordination with the patients nurse. When appointments need to be changed or moved, the therapist either completes the task or sends an in-basket message to schedulers. Generally, the PICU population includes children with a critical illness from birth to 18 years of age. Patients in the PICU are referred for OT and/or PT services by a physician or nurse practitioner via an electronic medical record consult order. There is a single inpatient OT and PT consult referral, therefore a scheduler places patients on the appropriate discipline schedule by matching the patients diagnoses and problems indicated by the referring provider against established criteria for distribution of OT/PT referrals at our institution. The PICU population included in this QI project consisted of patients with: an OT and PT consult, a scheduled OT and/or PT appointment, and a bed on the PICU floor. Exclusion criteria included: patients who do not have an OT and PT consult or the consult has been placed on hold. INCREASING UTILIZATION OF OT AND PT IN THE PICU 26 Procedures The primary investigator (PI) has participated in basic and intermediate improvement collaboratives designed to develop knowledge and expertise in the application of QI methodology. As an organizational leader, the PI has developed capability and is uniquely qualified to lead improvement work as evident by successful application of QI methodology as measured by statistically significant improvement of several RCIC projects, an intermediate improvement science series (I2S2) project, participation in advanced improvement leadership systems (AILS), and has served as a coach to other RCIC projects (Kaminski et al., 2014). The team included the PI who is the inpatient clinical manager, two physical therapists, one occupational therapist, one program assistant, and one PICU nurse. The QI project was guided by the RCIC roadmap (Figure 1). The RCIC roadmap is designed to guide the team through the process of applying the CCHMC QI methodology to their specific problem or project (Luzader, 2018). Prior to the start of the QI project all team members completed online modules intended to introduce staff to basic QI concepts, terminology and measurement. The RCIC process included structured learning time one time every three to six weeks delivered by the PI in consultation with RCIC leadership, weekly team meetings for 45-60 minutes, monthly feedback from senior leadership within the Division of OT and PT at CCHMC, and frequent 15-20 minute huddles during testing of interventions. The first step in the RCIC roadmap is to identify the problem. This is completed by identifying what is the problem, who does it affect, how often does it occur, why is the problem relevant, and why do we care? (Luzader, 2018). The problem identified in this project was the failure to deliver OT and PT services to patients in the PICU as determined through scheduling INCREASING UTILIZATION OF OT AND PT IN THE PICU 27 and service utilization data. The problem was vetted by the Division of OT and PT leadership and the staff directly involved in delivering OT and PT services in the PICU. The next step in the RCIC roadmap is documenting the current process for the management of daily OT and PT service delivery in the PICU. This is completed by documenting the current process, beginning with determining the beginning (start) and ending boundaries (stop) of the process. The team also considers the customer of the process and what the desired output or outcome is of the process. This was accomplished by brainstorming all existing action steps and decisions as a team in a single meeting. The team documented the current process based on each members personal experience with the process, placing action steps and decisions in chronological or sequential order. The team reviewed the action steps, combining those that were similar and expanding upon those that were not previously identified in sufficient detail. This resulted in a detailed process map that was expanded upon in subsequent weeks. This process map (Figure 2) was shared with other OT and PT staff who frequently provide services to the PICU for input and further modification in order to gain agreement on the existing process. This step took two to three weeks. By documenting the process in this manner, using consistent process map symbols (Figure A1), the team visually depicted the sequence of activities and decisions in the current process (Luzader, 2018). After documenting the current process, the team observed the process. The purpose of physically observing the action steps and decisions was to ensure the team agreed that the process map was truly representative of the process (Luzader, 2018). Once the existing process map for management of daily OT and PT services in the PICU was complete, this was accomplished by four out of five team members involved in the QI project observing the process. Over the course of two to three weeks the individuals first silently observed the process and then INCREASING UTILIZATION OF OT AND PT IN THE PICU 28 actively observed the process again, this time asking questions to gain clarity. An observation tool was developed using the detailed process map that team members used to document discrepancies, make additions to the process, document failures, and denote unexpected findings. The focus of the observation was on how the process works prior to any assessment of opportunities for improvement (Luzader, 2018). Following observations, the team edited the process map based upon additional learnings. The next step in the RCIC roadmap was to identify and analyze process failures. After confirming through observation that the current process map accurately captured the work and agreement was reached amongst those doing the work, the team identified, analyzed, and quantified failures. A high level process map with seven action steps was developed. This allowed the team to identify and analyze process failures using a simplified failures mode effect analysis (sFMEA) tool (Figure 3) over the course of two to three weeks. Any variation, inefficiencies, or rework were identified along with what was perceived to be working well (Luzader, 2018). The team engaged in a brainstorming process to identify actual and potential failures for each action step in the process (Luzader, 2018). The sFMEA tool helps illustrate both what has and what may go wrong at each action step along the process. Following creation of a sFMEA it was shared with other OT and PT staff who frequently provide services to the PICU for input and subsequent modification. The team chose a list of failures from the sFMEA to gather data on (Luzader, 2018). Lastly, the frequency of failures data or problems identified in the process were gathered and graphically represented on a Pareto chart (Figure 4). The data collected for each failure was summed and rank ordered in a Pareto chart from most frequent to least frequent (Langley et al., 2009; Luzader, 2018). These failures assisted in quantifying the INCREASING UTILIZATION OF OT AND PT IN THE PICU 29 relative importance of failures and was used to help guide the team in selecting which failure to focus on first (Langley et al., 2009; Luzader, 2018). The next step in the RCIC roadmap is to identify root causes for the most prevalent process failures previously identified. Starting with the failures with the highest frequency the team used the root cause analysis tool (Figure 5) by asking Why? at least 5 times (Langley et al., 2009; Luzader, 2018). Over several weeks the team answered why the failures occurred, the answers were written below the failure and the question repeated at least five times until the root cause of the failure was identified (Luzader, 2018). If the sequence of answers provided did not identify the root cause of the failure or problem, then the team started over. To verify if a root cause was identified the team started with the root cause and read in reverse order from the bottom to the top of the tool adding therefore to the end of the answer listed. If a root cause was appropriately identified, there should be a logical sequence when reading from the bottom up. This process was repeated for the top 4 failure modes until the root cause of the failures were identified. Determining the root-cause helped ensure that future interventions were directed at achieving the SMART Aim. After identifying the problem, documenting the current process, identifying and analyzing process failures, and identifying root causes, a SMART Aim including a measure was developed. A SMART Aim is specific, measurable, actionable, relevant, time bound, and population-specific (Luzader, 2018). The SMART Aim included the measure, a baseline and goal for which the QI team wanted to achieve, and a timeframe for achievement (Langley et al., 2009; Luzader, 2018). An operational definition is used to connect meaning to a concept in order to reduce variation in communication and measurement (Langley et al., 2009). The operational definition for this QI project defined the measure in the SMART Aim statement and ensured INCREASING UTILIZATION OF OT AND PT IN THE PICU 30 there was clarity & agreement on how data was being collected. This ensured the team had clarity and agreement on how to operationalize the measure and ensures repeatability of measures (Luzader, 2018). The SMART Aim measure was tracked daily during the weekday (Monday through Friday), with the exception of holidays. Data was tracked daily since the number of patients scheduled each day varies. Furthermore, data was tracked as a percentage in order to gain a better understanding of how many opportunities there were to deliver care (denominator) versus how many were actually provided (numerator). Data was tracked for at least four weeks prior to testing potential interventions in order to establish a baseline measure and set the goal for which the QI team wanted to achieve. Data was plotted on a run chart to graphical display the data over time (Langley et al., 2009). By establishing a baseline prior to testing interventions, the team would also be able to determine if the interventions tested were having the desired impact. The SMART Aim described the focus of the QI project (Luzader, 2018) and took approximately 4 weeks to complete. In the next step, the team identified key drivers which needed to be in place and working well in order to achieve the SMART Aim (Luzader, 2018). Key drivers were identified over two weeks of brainstorming all potential drivers. Ideas may come from examining failures listed on the sFMEA tool and turning them into positives (Luzader, 2018). The team clarified and combined drivers as appropriate. Key drivers were added or removed during the project based upon learnings. Key drivers were written as specific affirmative statements that include a noun and descriptor that serve to explain what is necessary to drive improvement (Luzader, 2018). All key drivers were assessed using the RCIC Assessment Tool to ensure all the components were present (Figure 6) (Luzader, 2018). INCREASING UTILIZATION OF OT AND PT IN THE PICU 31 The next step in the RCIC roadmap was to identify potential interventions that had a strong potential to eliminate root causes. Potential interventions to address process failure(s) that occurred most often were identified by reviewing key drivers, failures, root causes, and change concepts (Langley et al., 2009; Luzader, 2018). The team took approximately three weeks to generate ideas. The team began by reviewing key drivers, along with process failures listed on the sFMEA tool, the Pareto chart, and root cause tool, followed by brainstorming a list of possible interventions. Lastly the team reviewed a list of change concepts (Figure 7), considering which seem appropriate within the context of this project and identified additional possible interventions (Luzader, 2018). Once brainstorming was concluded the team confirmed potential interventions relate to one or more key drivers. Potential interventions were listed in the sFMEA tool (Figure 8) and prioritized based upon the predicted impact to the SMART Aim in order to choose which intervention to test first (Luzader, 2018). A key driver diagram (KDD) is used to schematically organize and display the SMART Aim, key drivers, and potential intentions, which combined represent our theory for improvement (Figure 9). The next step in the RCIC roadmap was to test the chosen interventions. PDSAs are the method used to test potential interventions. Interventions should be tested on a small scale, often starting with one patient, one therapist, or one day (Luzader, 2018). A PDSA ramp planning tool (Figure 10) was used to clearly depict the plan for a series of tests. PDSA ramps sequentially building on the knowledge gained from PDSAs (Luzader, 2018). Each test of change was designed using PDSA methodology and documented on a PDSA worksheet (Figure 11). The PDSA worksheet was used to record the plan, document what happened during the test, to record what was learned and what the team would do next based on the results of each test. The plan included the intervention being tested, the start and end date, what the objective of the particular INCREASING UTILIZATION OF OT AND PT IN THE PICU 32 test is, the test population, and which key driver it was intended to impact. Most importantly, it included how success was measured, a quantitative prediction, and a data collection plan (Luzader, 2018). After the test was executed the team huddled to discuss what occurred during the test and to decide next steps. Included in discussion of what occurred during the test was whether the test was carried out as planned, recording of data and observations, and identifying what occurred that was not part of the plan. The team studied whether the results matched the prediction, compared results to previous performance, and analyzed the results and document what the team learned. Following analysis of the results the team determined whether to adapt, adopt, or abandon the intervention and documented next steps for the intervention strategy (Luzader, 2018). Refinements and adjustments were made as needed in subsequent PDSA cycles (Luzader, 2018). PDSA cycles continued for at least 8 weeks or up until 120 days. After 60 days of PDSA cycles the Pareto was updated to determine whether there had been changes in previously identified failures. Lastly, as the interventions were tested through PDSA cycles, successful changes were strategically scaled across the target population as defined in the operational definition in a logical, well-thought-out manner (Langley et al., 2009; Luzader, 2018). Data Collection, Management, and Analysis The specific measure in the SMART Aim addressed in this RCIC project was tracked over time using a run chart (Figure 12). A run chart is a graphical display of data plotted in some type of order (Langley et al., 2009). Data was collected from the electronic medical record used at CCHMC which is EPIC (Verona, WI) as well as manual data collection when desired data could not be obtained via the electronic medical record. Data was plotted daily on the run chart and analyzed for special cause variation signaling a statistically significant change. Once a INCREASING UTILIZATION OF OT AND PT IN THE PICU 33 baseline median was established via the run chart, testing of interventions to achieve the SMART Aim began. The median was moved when a statistically significant change in the data occurred using the CCHMC Rules for Finding Special Causes (Figure 13). CCHMC Rules for Finding Special Causes are a set of rules that show patterns in the data that determine if the interventions led to the desired improvement. Once enough data had been collected the RCIC project was transitioned from a Run Chart to a Control Chart which is probability distribution based. Failure data were analyzed using a Pareto chart at the beginning of the QI project to quantify failures and after 60 days of testing to see if the frequencies & rank ordering on the Pareto Chart had changed signaling the elimination of a root cause. Results Data Baseline data were extracted over time from October 30, 2018 to December 6, 2018. The baseline data collection established the median percentage of scheduled OT and PT treatments delivered were 48.6% (Figure 12). The testing time period was 120 days during the time period of December 18, 2018 through April 16, 2019 and is presented in a run chart (Figure 12). Figure 12 shows the daily data collection results relative to the SMART Aim that occurred from October 30, 2018 to April 16, 2019. Once the baseline was established, the goal rate of OT and PT treatments which were delivered was set to be 58%, which was a 20% improvement. The SMART Aim therefore was to increase percent of scheduled OT and PT sessions which are delivered to patients in the PICU from 48% to 58% by April 16, 2019. Baseline data for OT and PT appointments in the PICU which were deferred were identified from October 24, 2018 to October 30, 2018. Deferrals occurred when a patient was scheduled for a therapy session, but the session was not initiated. Fifty-six deferred appointments INCREASING UTILIZATION OF OT AND PT IN THE PICU 34 were grouped into thirteen categories using a Pareto chart (Figure 4). The baseline data collection established that prior to the start of interventions, therapy was most frequency deferred because OT and PT were on hold for medical reasons, the patient was with another service when OT or PT attempted, the patient did not have an acute therapy need and for inappropriate timing of the OT/PT consult. Figure 4. Frequency & rank ordering of reasons patients were unavailable to receive OT/PT services in PICU from 10/24/18 to 10/30/18. Repeat data for OT and PT appointments in the PICU which were deferred were identified from March 11, 2019 to March 15, 2019. Sixty-three deferred appointments were grouped into the same thirteen categories using a Pareto chart (Figure 15). The repeat data collection established that following 60 days of interventions therapy was most frequency deferred because the patient was with another service when OT or PT attempted, the patient or INCREASING UTILIZATION OF OT AND PT IN THE PICU 35 family refused, the patient was undergoing a procedure when OT or PT attempted and because OT and PT was on hold for medical reasons. Figure 15. Frequency & rank ordering of reasons patients were unavailable to receive OT/PT services in PICU from 3/11/19 to 3/15/19. Statistical Analysis After the baseline median of 48.6% (Figure 12) was established and testing began, special cause rules (Figure 13 and Figure 14) were utilized to determine if the interventions led to the desired improvement. Run chart data plotted daily were analyzed for special cause variation signaling a statistically significant change. INCREASING UTILIZATION OF OT AND PT IN THE PICU 36 Failure data were analyzed using a Pareto chart at the beginning of the QI project to quantify failures (Figure 4) and after 60 days of testing (Figure 15) to see if the frequencies & rank ordering on the Pareto Chart had changed signaling the elimination of a root cause. PDSA Cycles During testing, 5 different interventions were tested, with a varying number of PDSA cycles (e.g., 36). One of these was ultimately abandoned, while 4 were adopted as useful intervention (Figure 16). Intervention 1 Plan-Do-Study-Act Intervention Communication with bedside nurse following initial attempt to request contact when patient is available for therapy. Summary of Outcomes Adopted. Therapist schedules do not support physically checking back on patients numerous times. Therapists collaborated with nursing to identify preferred mode of communication. A standard message was sent via internal messaging system. Communication was quick for therapist and allowed nurse to efficiently reply when patient was available. PDSA Cycles 4 Impact to Which Key Driver(s) Coordination with other providers and appropriate scheduling 2 Engage residents in identifying appropriate patients for consult to OT/PT. Adopted: Identified that therapists were unnecessarily over-processing orders sent for inappropriate patients. Multipronged approach included 1) statement in email sent to residents immediately prior to their PICU rotation along with a handout that includes indicators for appropriate consults; 2) Brief (15 min) learning session with OT/PT as part of resident onboarding during 1st week of rotation; 3) Handout including indicators for appropriate consults in resident call room. 5 Appropriate ID of patients for therapy consult 3 Schedule new consults for evaluation after medical team has completed their intake and history and physical (H&P). Adopted: Identified that OT/PT consult is part of the admission order set, therefore therapists were receiving consults and attempting to see patients before the medical team had finished their H&P. Referral management staff were provided criteria for which PICU consults should be scheduled for evaluation the following day for patient safety. Scheduler messaged therapist on Day 0 for awareness only. 6 Appropriate scheduling 4 Engage nursing in identifying a patients readiness for OT/PT interventions. Adopted: Identified that there was not agreement amongst nursing staff and therapists regarding when it was appropriate to involve OT/PT in rehabilitative interventions. To increase the ability for nursing to identify patients appropriateness for OT/PT services a presentation was delivered to nurse orientee's which included case-based scenarios . 3 Nursing identifies patient readiness and Buy-In of Unit staff regarding value of OT & PT services INCREASING UTILIZATION OF OT AND PT IN THE PICU 5 Determine number of patient on daily caseload that would allow therapist adequate time to see scheduled patients Abandoned: Unable to automate or efficiently adjust for fluctuations in available work hours per therapist therefore it continued to be a manual process. 37 5 Appropriate scheduling and adequate time for OT & PT to see scheduled patients Figure 16: PDSA Interventions and Outcomes are listed the twenty-three specific PDSAs that were tested are described. Analyzing the Run Chart/Control Chart Improvement in the primary outcome measure was demonstrated when special cause variation signaled a statistically significant change. After thirty-two days of testing the center line shifted upward. The new median rate of OT and PT treatments delivered was calculated (February 4, 2019-February 13, 2019), for a value of 66%. This represents a 37.5% increase in the number of scheduled OT and PT treatments which were delivered. This increase was not sustained and after eight more days of testing the center line shifted down. The new median rate of scheduled OT and PT treatments delivered was calculated (February 14 February 25), for a value of 56%. This represented a subsequent 15.2% decrease in the number of scheduled OT and PT treatments which were delivered. The percentage of scheduled OT and PT treatments delivered fluctuated over the course of this QI project (December 18, 2018 April 16, 2019). From a baseline of 48%, there were two shifts before stabilizing at 56% of scheduled OT and PT treatments being delivered on the day scheduled, representing an overall increase of 16.7%. Data was transferred to a statistical process control chart to identify whether changes in processes were making a difference in the outcomes over time using a P-chart, which is an attribute control chart (Figure 17). The P-chart indicated that the mean improved from 48.6% to 56% at the end of the time period. Since the processes had clearly changed, new control limits reflected a desired decrease in day-to-day variation (Figure 17). INCREASING UTILIZATION OF OT AND PT IN THE PICU 38 Improvement in the secondary measures was demonstrated through change in the frequency & rank ordering of reasons patients were unavailable for scheduled OT and PT appointments in the PICU. Changes in frequencies & rank ordering on the Pareto Chart signal a decrease in frequency of the following root causes (Figure 18): Therapy placed on hold Patient did not have any acute therapy needs Inappropriate timing of consult Therapy placed on hold for social reasons Patient too sedated to participate in OT or PT Patient transferring units Patient already in chair Patient discharging INCREASING UTILIZATION OF OT AND PT IN THE PICU 39 Figure 18. Comparison of frequency & rank ordering of reasons patients were unavailable to receive OT/PT services in PICU. October 24, 2018 to October 30, 2018 compared to March 11, 2019 to March 15, 2019. The change in frequency of reasons patients were unavailable signaled a decrease in root causes. INCREASING UTILIZATION OF OT AND PT IN THE PICU 40 Figure 12. Run chart indicating the percentage of Patients Scheduled for OT or PT Who Received Services in the PICU from October 30, 2018 - April 16, 2019. The center line shifted from 48.6% to 66% on the run chart, then shifted to 56% and remained there at the end of the QI project. With a 16.7% improvement the center line did not reach the 20% improvement goal. The observed data are indicated by the blue line with markers. The green arrow in the upper righthand corner represents the desired direction of change. The center line is indicated by the solid red line and represents the overall median score for a specified period of time. This percent change is calculated using the formula (V2 V1)/V1 * 100, where V1 is equal to the initial value and V2 is equal to the final value. INCREASING UTILIZATION OF OT AND PT IN THE PICU 41 Figure 17. P-chart indicating by day the percent of patients Scheduled for OT or PT Who Received Services in the PICU from October 30, 2018 - April 16, 2019. The mean shifted from 48.6% to 66% on the p-chart, then shifted to 56% and remained there at the end of the QI project. With a 16.7% improvement the mean did not reach the 20% improvement goal. The observed data are indicated by the blue markers. The green arrow in the upper right-hand corner represents the desired direction of change. The mean is indicated by the solid red line and represents the overall average score for a specified period of time. The control limits, indicated by the dashed red lines, set at +3SD, are used to determine when the process is out of control. INCREASING UTILIZATION OF OT AND PT IN THE PICU 42 Discussion Current barriers negatively impacted the delivery OT and PT services to patients in the PICU at our institution. The purpose of this study was to assess whether the desired improvement in the delivery of OT and PT services in the PICU was achieved when interventions were applied using QI methodology. Data collected by comparing how many opportunities there were to deliver care versus how many were provided were analyzed to inform whether interventions applied during a 120 day QI project were effective in transforming the care. Using QI methods, we increased the percentage of scheduled OT and PT treatments which were delivered in the PICU by 16.7%, which was short of our 20% improvement goal. Additionally, the study provided insights into the reasons appointments were deferred at our institution in order to prioritize interventions on the problems which have the greatest potential to impact the delivery of OT and PT treatments. Our project built on previous work that developed a proactive system to increase consults for OT and PT services within 72 hours of admission. Because we wanted to improve the ability to deliver OT and PT services in our institution, we did not predetermine what interventions to test based upon findings in the literature. Rather, we collected data on the root causes for why OT and PT services were deferred at our institution and rank ordered them by highest to lowest frequency to effectively prioritize interventions. We observed the most frequent reasons appointments were deferred at the start of the project were when nursing declined OT and PT services for medical reasons (21.4%); when another provider was with the patient (16.1%); when patients did not have any acute therapy needs, such as an admission for bronchiolitis with an expected length of stay < 72 hrs, as determined during chart review (16.1%); and when the timing of consult was inappropriate (8.9%), for example submitted by a PICU resident before INCREASING UTILIZATION OF OT AND PT IN THE PICU 43 surgical restrictions affecting mobility were discontinued. Patient level barriers such as medical instability were perceived by 90.9% of providers to be a barrier to EM (Choong et al., 2013). Although 90.9% of providers surveyed perceived medical instability was a barrier, Choong et al. (2013) did not assess the frequency at which this barrier occurred as was done in our study and Wieczorek et al. (2016). Our results agree with the findings of Wieczorek et al. (2016) who reported frequent reasons for deferral were due to medical instability (16% and 22% respectively) and procedures (16% and 20% respectively). In addition, we experienced a high frequency of deferrals at the request of nursing (21%). These results are similar to the findings of Fink et al. (2019) and Cui et al. (2017) who reported a frequent reason for deferral of therapies was at the request of nursing (18% for PT and 16% for OT and 50% for OT and PT combined respectively). A key driver to increase the percentage of scheduled OT and PT sessions which were delivered to patients in the PICU was efficient coordination with other providers. In a survey of provider beliefs, Joyce et al. (2018) found that therapists report time constraints and increased workload as barriers to the delivery of EM in the PICU. Although Joyce et al. (2018) did not specify the type of time constraints nor propose solutions, we observed that therapist were unable to physically return to the patients room numerous times a day when the patient was unavailable on the first attempt. Therefore, we tested novel methods for coordination with the bedside nurse that would alert OT or PT when the patient was available after the first attempt. We used both low reliability methods such as patient specific reminder cards and posting reminders on patient doors and higher reliability methods such as standardized text messaging. Collaboratively, nursing and therapy staff determined that a text message was the preferred method of communication. Therapists created a standard message that was efficiently inserted into a text, INCREASING UTILIZATION OF OT AND PT IN THE PICU 44 individualized to the patient, and then sent to the bedside nurse. Nursing reported it was simple and efficient to reply back to that message once the patient was available for OT or PT. Another key driver was appropriate identification of patients who would benefit from OT and PT services. A failure we observed was therapists engaging in unnecessary chart reviews and communication with nursing as a result of new consult orders received for patients who did not have an acute therapy need. Educational methods were tested to assist residents in appropriate identifications of patients who would benefit from OT and PT services. One challenge we faced was maintaining a system during turnover of resident providers. To overcome this obstacle, we used both low reliability methods such as education and training and higher reliability methods such as decision aids. We learned that integrating information about OT and PT services via existing communication paths when onboarding residents was beneficial. This multipronged approach included a statement in a newsletter sent immediately preceding the PICU rotation along with an attached PDF that included indicators for appropriate consults. The same PDF was posted in the resident call room, and a brief (15 min) interactive learning session with OT and/or PT was integrated into the first week of a PICU rotation. We also discovered that repeating the set of 4 interventions with each change in team composition improved the reliability that consult orders placed were for patients who had an acute therapy need. By anticipating when new residents would arrive and implementing our set of interventions at the start of their rotation, we improved and maintained appropriate OT/PT consult orders through multiple changes in staff. We observed that the education bundle reduced the number of consults received for patients who did not have an acute therapy need (Figure 18). We observed that this challenge with inappropriate OT/PT consult orders was the result of orders being placed at the time of admission. This resulted in therapists arriving to see patients INCREASING UTILIZATION OF OT AND PT IN THE PICU 45 prior to the medical team ascertaining if the patient was clinically safe to participate in therapy interventions. Other studies have found that consults were received 2 or more days after admission to the PICU or considered delayed (Choong et al., 2013; Cui et al., 2017; Wieczorek et al., 2016), thus this was a unique challenge to our institution. We tested methods to schedule patients appropriately, which has previously been identified as another key driver to success in this QI project. Scheduling staff were provided criteria for which PICU consult orders should be scheduled for evaluation the following day. Schedulers were instructed to message the therapist on Day 0 for awareness, but therapies would not be initiated until Day 1 of admission to the PICU. Obtaining orders within 24 hours of admission as recommended by Choong, Canci, et al. (2018) was typical at our institution following previous QI initiatives. Additional key drivers we focused on were nursing identification of patient readiness and buy-in of unit staff regarding the value of OT & PT services. Similar to other studies, we observed lack of agreement amongst staff regarding patient suitability to participate in rehabilitative interventions (Choong et al., 2013; Wieczorek et al., 2015). In addition, studies have suggested that PICU nursing frequently deferred OT and PT sessions (Choong et al., 2013; Cui et al., 2017; Fink et al., 2019). We discovered that knowledge regarding which patients may benefit from OT or PT services and what interventions OT and PT provide to critically ill children varied amongst nursing staff at our intuition. A recently published practice recommendation proposes that the decision to provide mobility interventions should involve an interprofessional team, including occupational therapists and physical therapists who have expertise in mobilizing patients (Choong, Canci, et al., 2018). Therefore, we focused on educational methods that would emphasize the breadth of therapy interventions available to critically ill patients. Initial PDSAs focused on establishing agreement between novice nurses INCREASING UTILIZATION OF OT AND PT IN THE PICU 46 and a therapist or an experienced nurse using current patients to determine readiness for OT and PT. Following several PDSA cycles to establish agreement, case-based education was developed by the QI team. Lastly, it was delivered during an interactive nursing orientation session in order to facilitate dialog around accurate identification of patients ready to participate in OT and PT interventions. A 6% decrease in the frequency of nursing staff deferring therapy was observed (Figure 18). Lastly, adequate time for OT and PT to see scheduled patients was identified as a key driver to success in this QI project. It was reported that therapists workload at our institution may limit their ability to serve patients who were receiving care from another provider at the time they attempted to provide OT or PT services. Other studies have also found therapists report time constraints, increased staff workload (Joyce et al., 2018) and staffing limitations (Choong et al., 2013) as barriers to providing OT and PT services. We learned that determining a reasonable number of scheduled patients based on hours available was achievable. Despite some initial success this intervention involved a labor intensive process that requires therapists to remember to manually denote available hours in an unused field within the electronic scheduling system weekly. After a few PDSA cycles therapists reported this manual process was not sustainable within their current workload, therefore, this intervention was abandoned. Leveraging the existing QI culture associated with increasing OT and PT services for the purposes of early mobilization at our institution, we found the interprofessional team to be receptive to PDSA cycles. We tested several methods of communication with bedside nurses, educated residents on appropriate identification of patients who would benefit from therapy, proactively scheduled initial evaluations when patients were most likely to be available, educated nurses to identify patients readiness for OT and PT services, and determined the number of INCREASING UTILIZATION OF OT AND PT IN THE PICU 47 patients a therapist could manage on a daily basis. Although we fell short of our 20% improvement goal, the QI team reported that engaging in a series of PDSAs to address and mitigate many system issues linked to therapists workload and collaborative relationships amongst therapists, nurses and residents was valuable. We hypothesized that the high acuity of patients in the PICU made it difficult to predict their course of care and was a barrier to achieving our goal. In addition, we hypothesized that the lack of routine modes of communication for interprofessional collaboration made it challenging to anticipate when patients were most likely to be available for therapy or when other procedures or services were anticipated on the same day. Furthermore, order entry using an order set combined with resident turnover were barriers to achieving our goal. Based on a number of observations, residents tended to choose a generic order set that included OT and PT even when a patients presentation did not indicate a need for therapy services. This added to a therapists workload and adversely affected the time therapists had available to provide interventions to patients in need of OT or PT services. Lastly, the 37.5% increase followed by a subsequent 15.2% decrease in the measure is believed to be the result of interventions to reduce the number of consults received for patients who did not have an acute therapy needs in addition to successful interventions to schedule patients starting the day after admission. It is not uncommon for a team to see a significant bump in their measure as a result of increased effort and persistence by early adopters only to have it stabilize at a lower level. Limitations This study is subject to a number of limitations. First, the study was conducted at a single health care institution by a small QI team consisting of the PI, two physical therapists, one occupational therapist, one program assistant, and one PICU nurse. As such, the results may not INCREASING UTILIZATION OF OT AND PT IN THE PICU 48 be generalizable to other institutions. Second, our project relied heavily on a combination of EMR and manual end of day self-reporting to assist in data collection to identify which scheduled OT and PT treatments were delivered. We believe that an automated system for inclusion of eligible appointments is likely more effective and objective than self-reporting, and implementation in a system without an EMR or dedicated PICU staff may require significant effort. Finally, the goal of our project was to drive rapid change in practice. The QI methodology was determined to be the best method to achieve this goal. As such, the specific mechanism that resulted in change could not be identified. Implications for Practice and Future Research Although we sustained our improvement for 2 months, we did not reach our goal of a 20% improvement in the percentage of scheduled OT and PT treatments which were delivered in the PICU. Although the results may not be generalizable, the experiences of therapist workload challenges, systems and processes which result in processing inappropriate consult orders, frequent resident turnover, and lack of agreement amongst staff regarding patient suitability to participate in rehabilitative interventions exists in many pediatric health care institutions. Given the generic nature of these interventions, they could likely be replicated in other care environments. For those who chose to replicate these interventions in other health care institutions, spread concepts should be used to ensure success (Langley et al., 2009). Our work was not able to address the issues regarding inefficient coordination with other providers or the availability of appropriate staff and equipment. In addition, despite several PDSAs, we were unable to automate our scheduling system to support scheduling a reasonable number of patients based upon available work hours per therapist. Future work may look to better address these challenge to determine their potential effect on further improvement. In addition, the INCREASING UTILIZATION OF OT AND PT IN THE PICU 49 interventions applied in this QI project are being spread to other occupational therapy and physical therapy staff at our institution who are most likely to provide coverage in the PICU. Conclusion This study described how QI methodologies and tools facilitated in a structured, organized manner at our institution promoted the improvement of care delivery processes. We successfully identified and applied interventions during a focused, narrow-scoped 120 day QI project that were effective in improving the care delivery processes for therapy services in the PICU as measured by an improvement in the percentage of scheduled OT and PT treatments which are delivered. Key drivers of success for our QI project included (a) appropriate identification of patients for OT/PT consult orders, (b) nursing identification of patient readiness, (c) appropriate scheduling, (d) buy-in of unit staff regarding the value of OT and PT services, and (e) efficient coordination with other providers. Many of the key drivers and interventions that were used in this project are generalizable to other units within our institution and given their generic nature, may be replicated elsewhere. INCREASING UTILIZATION OF OT AND PT IN THE PICU 50 References Abdulsatar, F., Walker, R. G., Timmons, B. W., & Choong, K. (2013). "Wii-Hab" in critically ill children: A pilot trial. Journal of Pediatric Rehabilitation Medicine, 6(4), 193-204. doi:10.3233/PRM-130260 Adler, J., & Malone, D. (2012). Early mobilization in the intensive care unit: A systematic review. Cardiopulmonary Physical Therapy Journal, 23(1), 5-13. Aspesberro, F., Mangione-Smith, R., & Zimmerman, J. J. (2015). Health-related quality of life following pediatric critical illness. Intensive Care Medicine, 41(7), 1235-1246. doi:10.1007/s00134-015-3780-7 Baily, M. A., Bottrell, M., Lynn, J., Jennings, B., & Hastings, C. (2006). The ethics of using QI methods to improve health care quality and safety. The Hastings Center Report, 36(4), S1-40. Betters, K. A., Hebbar, K. B., Farthing, D., Griego, B., Easley, T., Turman, H., . . . deAlmeida, M. L. (2017). Development and implementation of an early mobility program for mechanically ventilated pediatric patients. Journal of Critical Care, 41, 303-308. doi:10.1016/j.jcrc.2017.08.004 Billett, A. L., Colletti, R. B., Mandel, K. E., Miller, M., Muething, S. E., Sharek, P. J., & Lannon, C. M. (2013). Exemplar pediatric collaborative improvement networks: achieving results. Pediatrics, 131 Suppl 4, S196-203. doi:10.1542/peds.2012-3786F Boyd, A. (2016). Increasing musculoskeletal screening in a hospital-based pediatric weight management program: A quality improvement initiative. (Unpublished doctoral project), University of Indianapolis, Indianapolis, IN. INCREASING UTILIZATION OF OT AND PT IN THE PICU 51 Cameron, S., Ball, I., Cepinskas, G., Choong, K., Doherty, T. J., Ellis, C. G., . . . Fraser, D. D. (2015). Early mobilization in the critical care unit: A review of adult and pediatric literature. Journal of Critical Care, 30(4), 664-672. doi:10.1016/j.jcrc.2015.03.032 Choong, K., Awladthani, S., Khawaji, A., Clark, H., Borhan, A., Cheng, J., . . . Al-Harbi, S. (2017). Early exercise in critically ill youth and children, a preliminary evaluation: The wEECYCLE pilot trial. Pediatric Critical Care Medicine, 18(11), e546-e554. doi:10.1097/pcc.0000000000001329 Choong, K., Canci, F., Clark, H., Hopkins, R. O., Kudchadkar, S. R., Lati, J., . . . Zebuhr, C. (2018). Practice recommendations for early mobilization in critically ill children. Journal of Pediatric Intensive Care, 07(01), 014-026. doi:10.1055/s-0037-1601424 Choong, K., Chacon, M. D. P., Walker, R. G., Al-Harbi, S., Clark, H., Al-Mahr, G., . . . Thabane, L. (2015). In-bed mobilization in critically ill children: A safety and feasibility trial. Journal of Pediatric Intensive Care, 4(4), 225. Choong, K., Foster, G., Fraser, D. D., Hutchison, J. S., Joffe, A. R., Jouvet, P. A., . . . Ward, R. E. (2014). Acute rehabilitation practices in critically ill children: a multicenter study. Pediatric Critical Care Medicine, 15(6), e270-279. doi:10.1097/pcc.0000000000000160 Choong, K., Fraser, D., Al-Harbi, S., Borham, A., Cameron, J., Cameron, S., . . . Thabane, L. (2018). Functional recovery in critically ill children, the "WeeCover" multicenter study. Pediatric Critical Care Medicine, 19(2), 145-154. doi:10.1097/pcc.0000000000001421 Choong, K., Koo, K. K., Clark, H., Chu, R., Thabane, L., Burns, K. E., . . . Meade, M. O. (2013). Early mobilization in critically ill children: a survey of Canadian practice. Critical Care Medicine, 41(7), 1745-1753. doi:10.1097/CCM.0b013e318287f592 INCREASING UTILIZATION OF OT AND PT IN THE PICU 52 Cincinnati Children's Hospital Medical Center. (2018). Pediatric Intensive Care Unit (PICU). Retrieved from https://www.cincinnatichildrens.org/service/p/picu/hcp Clauss, S. B., Anderson, J. B., Lannon, C., Lihn, S., Beekman, R. H., Kugler, J. D., & Martin, G. R. (2015). Quality improvement through collaboration: the National Pediatric Quality improvement Collaborative initiative. Current Opinion in Pediatrics, 27(5), 555-562. doi:10.1097/MOP.0000000000000263 Colwell, B. R. L., Olufs, E., Zuckerman, K., Kelly, S. P., Ibsen, L. M., & Williams, C. N. (2019). PICU Early Mobilization and Impact on Parent Stress. Hospital Pediatrics, 9(4), 265272. doi:10.1542/hpeds.2018-0155 Cremer, R., Leclerc, F., Lacroix, J., & Ploin, D. (2009). Children with chronic conditions in pediatric intensive care units located in predominantly French-speaking regions: Prevalence and implications on rehabilitation care need and utilization. Critical Care Medicine, 37(4), 1456-1462. doi:10.1097/CCM.0b013e31819cef0c Cuello-Garcia, C. A., Mai, S. H. C., Simpson, R., Al-Harbi, S., & Choong, K. (2018). Early Mobilization in Critically Ill Children: A Systematic Review. The Journal of Pediatrics, 203, 25-33.e26. doi:10.1016/j.jpeds.2018.07.037 Cui, L. R., LaPorte, M., Civitello, M., Stanger, M., Orringer, M., Casey, F., 3rd, . . . Fink, E. L. (2017). Physical and occupational therapy utilization in a pediatric intensive care unit. Journal of Critical Care, 40, 15-20. doi:10.1016/j.jcrc.2017.03.003 Dandoy, C. E., Hariharan, S., Weiss, B., Demmel, K., Timm, N., Chiarenzelli, J., . . . Alessandrini, E. (2016). Sustained reductions in time to antibiotic delivery in febrile immunocompromised children: results of a quality improvement collaborative. BMJ Quality & Safety, 25(2), 100-109. doi:10.1136/bmjqs-2015-004451 INCREASING UTILIZATION OF OT AND PT IN THE PICU 53 Deming, W. E. (2000). The new economics: For industry, government, education (2nd ed.). Cambridge, MA: MIT Press. Edwards, J. D., Houtrow, A. J., Vasilevskis, E. E., Rehm, R. S., Markovitz, B. P., Graham, R. J., & Dudley, R. A. (2012). Chronic conditions among children admitted to U.S. PICUs: their prevalence and impact on risk for mortality and prolonged length of stay. Critical Care Medicine, 40(7), 2196-2203. doi:10.1097/CCM.0b013e31824e68cf Fan, E. (2012). Critical illness neuromyopathy and the role of physical therapy and rehabilitation in critically ill patients. Respiratory Care, 57(6), 933-944; discussion 944-936. doi:10.4187/respcare.01634 Fink, E. L., Beers, S. R., Houtrow, A. J., Richichi, R., Burns, C., Doughty, L., . . . Kochanek, P. (2019). Early Protocolized Versus Usual Care Rehabilitation for Pediatric Neurocritical Care Patients: A Randomized Controlled Trial. Pediatric Critical Care Medicine, 20(6), 540-550. doi:10.1097/pcc.0000000000001881 Hawkins, E., & Jones, A. (2015). What is the role of the physiotherapist in paediatric intensive care units? A systematic review of the evidence for respiratory and rehabilitation interventions for mechanically ventilated patients. Physiotherapy, 101(4), 303-309. doi:10.1016/j.physio.2015.04.001 Hollander, S. A., Hollander, A. J., Rizzuto, S., Reinhartz, O., Maeda, K., & Rosenthal, D. N. (2014). An inpatient rehabilitation program utilizing standardized care pathways after paracorporeal ventricular assist device placement in children. The Journal of Heart and Lung Transplantation, 33(6), 587-592. doi:10.1016/j.healun.2013.12.009 INCREASING UTILIZATION OF OT AND PT IN THE PICU 54 Hopkins, R. O., Choong, K., Zebuhr, C. A., & Kudchadkar, S. R. (2015). Transforming PICU culture to facilitate early rehabilitation. Journal of Pediatric Intensive Care, 4(4), 204211. doi:10.1055/s-0035-1563547 Joyce, C. L., Taipe, C., Sobin, B., Spadaro, M., Gutwirth, B., Elgin, L., . . . Traube, C. (2018). Provider beliefs regarding early mobilization in the pediatric intensive care unit. Journal of Pediatric Nursing, 38, 15-19. doi:10.1016/j.pedn.2017.10.003 Kaminski, G. M., Schoettker, P. J., Alessandrini, E. A., Luzader, C., & Kotagal, U. (2014). A comprehensive model to build improvement capability in a pediatric academic medical center. Academic Pediatrics, 14(1), 29-39. doi:DOI 10.1016/j.acap.2013.02.007 Krmpotic, K., & Lobos, A. T. (2013). Clinical profile of children requiring early unplanned admission to the PICU. Hospital Pediatrics, 3(3), 212-218. Kukreti, V., Shamim, M., & Khilnani, P. (2014). Intensive care unit acquired weakness in children: Critical illness polyneuropathy and myopathy. Indian Journal of Critical Care Medicine, 18(2), 95-101. doi:10.4103/0972-5229.126079 Langley, G. J., Moen, R. D., Nolan, K. M., Nolan, T. W., Norman, C. L., & Provost, L. P. (2009). The improvement guide : A practical approach to enhancing organizational performance. San Francisco, CA: Jossey-Bass. Lee, C. S., & Larson, D. B. (2014). Beginner's guide to practice quality improvement using the model for improvement. Journal of the American College of Radiology, 11(12 Pt A), 1131-1136. doi:10.1016/j.jacr.2014.08.033 Luzader, C. P. (2018). Rapid cycle improvement collaborative. Paper presented at the Rapid Cycle Improvement Collaborative 30, Cincinnati Children's Hospital. INCREASING UTILIZATION OF OT AND PT IN THE PICU 55 Miura, S., Wieczorek, B., Lenker, H., & Kudchadkar, S. R. (2018). Normal baseline Function is associated with delayed rehabilitation in critically ill children. Journal of Intensive Care Medicine, 885066618754507. doi:10.1177/0885066618754507 Needham, D. M., Korupolu, R., Zanni, J. M., Pradhan, P., Colantuoni, E., Palmer, J. B., . . . Fan, E. (2010). Early physical medicine and rehabilitation for patients with acute respiratory failure: A quality improvement project. Archives of Physical Medicine and Rehabilitation, 91(4), 536-542. doi:10.1016/j.apmr.2010.01.002 Nolan, T., Resar, R., Griffin, F., & Gordon, A. B. (2004). Improving the reliability of health care. In IHI Innovation Series white paper. Boston, MA: Institute for Health Care Improvement. Retrieved from http://www.ihi.org/resources/Pages/IHIWhitePapers/ImprovingtheReliabilityofHealthCar e.aspx. Odetola, F. O. (2014). Pediatric Critical Care Medicine. In D. Wheeler, H. Wong, & T. Shanley (Eds.), Outcomes Research in the PICU (2 ed., Vol. 1, pp. 107-116). London: SpringerVerlag. Odetola, F. O., Clark, S. J., Freed, G. L., Bratton, S. L., & Davis, M. M. (2005). A national survey of pediatric critical care resources in the United States. Pediatrics, 115(4), e382386. doi:10.1542/peds.2004-1920 Ong, C., Lee, J. H., Leow, M. K., & Puthucheary, Z. A. (2016). Functional outcomes and physical impairments in pediatric critical care survivors: A scoping review. Pediatric Critical Care Medicine, 17(5), e247-259. doi:10.1097/pcc.0000000000000706 INCREASING UTILIZATION OF OT AND PT IN THE PICU 56 Owens, T., & Tapley, C. (2018). Pediatric Mobility: The Development of Standard Assessments and Interventions for Pediatric Patients for Safe Patient Handling and Mobility. Critical Care Nursing Quarterly, 41(3), 314-322. doi:10.1097/cnq.0000000000000210 Parisien, R. B., Gillanders, K., Hennessy, E. K., Herterich, L., Saunders, K., Lati, J., . . . O'Brien, K. K. (2016). Experiences of four parents with physical therapy and early mobility of their children in a pediatric critical care unit: A case series. Journal of Pediatric Rehabilitation Medicine, 9(2), 159-168. doi:10.3233/prm-160374 Parker, M. W., Carroll, M., Bolser, B., Ballinger, J., Brewington, J., Campanella, S., . . . Brady, P. W. (2017). Implementation of a Communication Bundle for High-Risk Patients. Hospital Pediatrics, 7(9), 523-529. doi:10.1542/hpeds.2016-0170 Piva, T. C., Ferrari, R. S., & Schaan, C. W. (2019). Early mobilization protocols for critically ill pediatric patients: systematic review. The Revista Brasileira de Terapia Intensiva, 31(2), 248-257. doi:10.5935/0103-507x.20190038 Pollack, M. M., Holubkov, R., Funai, T., Clark, A., Berger, J. T., Meert, K., . . . Jenkins, T. L. (2014). Pediatric intensive care outcomes: Development of new morbidities during pediatric critical care. Pediatric Critical Care Medicine, 15(9), 821-827. doi:10.1097/pcc.0000000000000250 Pollack, M. M., Holubkov, R., Funai, T., Clark, A., Moler, F., Shanley, T., . . . Jenkins, T. L. (2014). Relationship between the functional status scale and the pediatric overall performance category and pediatric cerebral performance category scales. JAMA Pediatrics, 168(7), 671-676. doi:10.1001/jamapediatrics.2013.5316 INCREASING UTILIZATION OF OT AND PT IN THE PICU 57 Randolph, A. G., Gonzales, C. A., Cortellini, L., & Yeh, T. S. (2004). Growth of pediatric intensive care units in the United States from 1995 to 2001. The Journal of Pediatrics, 144(6), 792-798. doi:https://doi.org/10.1016/j.jpeds.2004.03.019 Schaffzin, J. K., Prichard, H., Bisig, J., Gainor, P., Wolfe, K., Solan, L. G., . . . McCarthy, J. J. (2013). A collaborative system to improve compartment syndrome recognition. Pediatrics, 132(6), e1672-1679. doi:10.1542/peds.2013-1330 Schouten, L. M., Hulscher, M. E., van Everdingen, J. J., Huijsman, R., & Grol, R. P. (2008). Evidence for the impact of quality improvement collaboratives: systematic review. BMJ, 336(7659), 1491-1494. doi:10.1136/bmj.39570.749884.BE Schweickert, W. D., Pohlman, M. C., Pohlman, A. S., Nigos, C., Pawlik, A. J., Esbrook, C. L., . . . Kress, J. P. (2009). Early physical and occupational therapy in mechanically ventilated, critically ill patients: a randomised controlled trial. The Lancet, 373(9678), 1874-1882. doi:https://doi.org/10.1016/S0140-6736(09)60658-9 Strenk, M. L., Kiger, M., Hawke, J. L., Mischnick, A., & Quatman-Yates, C. (2017). Implementation of a Quality Improvement Initiative: Improved Congenital Muscular Torticollis Outcomes in a Large Hospital Setting. Physical Therapy, 97(6), 649-658. doi:10.1093/ptj/pzx029 Tsuboi, N., Hiratsuka, M., Kaneko, S., Nishimura, N., Nakagawa, S., Kasahara, M., & Kamikubo, T. (2019). Benefits of Early Mobilization After Pediatric Liver Transplantation. Pediatric Critical Care Medicine, 20(2), e91-e97. doi:10.1097/pcc.0000000000001815 INCREASING UTILIZATION OF OT AND PT IN THE PICU 58 Tsuboi, N., Nozaki, H., Ishida, Y., Kanazawa, I., Inamoto, M., Hayashi, K., . . . Kamikubo, T. (2017). Early Mobilization after Pediatric Liver Transplantation. Journal of Pediatric Intensive Care, 6(3), 199-205. doi:10.1055/s-0036-1593387 Turner, D. A., Cheifetz, I. M., Rehder, K. J., Williford, W. L., Bonadonna, D., Banuelos, S. J., . . . Zaas, D. (2011). Active rehabilitation and physical therapy during extracorporeal membrane oxygenation while awaiting lung transplantation: A practical approach. Critical Care Medicine, 39(12), 2593-2598. Van Damme, D., Flori, H., & Owens, T. (2018). Development of Medical Criteria for Mobilizing a Pediatric Patient in the PICU. Critical Care Nursing Quarterly, 41(3), 323-329. doi:10.1097/cnq.0000000000000212 Wieczorek, B., Ascenzi, J., Kim, Y., Lenker, H., Potter, C., Shata, N. J., . . . Kudchadkar, S. R. (2016). PICU Up!: Impact of a quality improvement intervention to promote early mobilization in critically ill children. Pediatric Critical Care Medicine, 17(12), e559e566. doi:10.1097/pcc.0000000000000983 Wieczorek, B., Burke, C., Al-Harbi, A., & Kudchadkar, S. R. (2015). Early mobilization in the pediatric intensive care unit: A systematic review. Journal of Pediatric Intensive Care, 2015, 129-170. doi:10.1055/s-0035-1563386 World Health Organization. (2007). International Classification of Functioning, Disability, and Health: Children & Youth Version: ICF-CY. Retrieved from http://apps.who.int/iris/bitstream/10665/43737/1/9789241547321_eng.pdf Zheng, K., Sarti, A., Boles, S., Cameron, S., Carlisi, R., Clark, H., . . . Choong, K. (2018). Impressions of Early Mobilization of Critically Ill Children-Clinician, Patient, and INCREASING UTILIZATION OF OT AND PT IN THE PICU Family Perspectives. Pediatric Critical Care Medicine, 19(7), e350-e357. doi:10.1097/pcc.0000000000001547 59 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figures Figure 1: RCIC Roadmap to Guide Teams through Application of Improvement Methodology (Luzader, 2018) Figure 2. Process Map which visually depicted the sequence of activities and decisions in the current process. 60 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 3. Simplified FMEA Diagram used to Identify Current Failures 61 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 5. Root Cause Analysis Worksheet used To Identify Future Interventions Figure 8. Simplified FMEA Diagram used to Identify Potential Interventions 62 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 9. Key Driver Diagram Relating SMART Aim to Key Drivers and Interventions 63 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 6. RCIC Assessment tool to ensure SMART Aim, Key Drivers, Run Charts, PDSAs & Ramps meet all criteria (Luzader, 2018) 64 INCREASING UTILIZATION OF OT AND PT IN THE PICU 65 Figure 7. Change concepts to consider when designing interventions in Luzader (2018), adapted from Langley et al. (2009) INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 10. PDSA Ramp Planning Tool Which will Combine Change Common to a Single Theme (Luzader, 2018). Figure 11. PDSA Worksheet That Will be Used for Each Test (Luzader, 2018). 66 INCREASING UTILIZATION OF OT AND PT IN THE PICU Figure 13. Special Cause Rules Applied to a Run Chart (Luzader, 2018) Figure 14. Special Cause Rules Applied to a Control Chart (Luzader, 2018). 67 INCREASING UTILIZATION OF OT AND PT IN THE PICU 68 INCREASING UTILIZATION OF OT AND PT IN THE PICU 69 Appendices Figure A1. Process map symbols used to visually depict the sequence of activities and decisions in a process (Luzader, 2018).  ...

... DETERMINING THE RELATIONSHIP BETWEEN LEVEL I FIELDWORK PERFORMANCE, LEVEL II FIELDWORK PERFORMANCE, AND COMPETENCE EXAM SUCCESS Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Jayson Zeigler, MS, OTR Copyright April 15, 2020 By: Jayson Zeigler, MS, OTR All rights reserved Approved by: Stephanie Kelly, PT, PhD Committee Chair ____________________________________ Elizabeth D. DeIuliis, OTD, MOT, OTR/L, CLA Committee Member ____________________________________ Elizabeth S. Moore, PhD Committee Member ____________________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis ____________________________________ Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis ____________________________________ Running head: FIELDWORK RELATIONSHIPS 1 Determining the Relationship Between Level I Fieldwork Performance, Level II Fieldwork Performance, and Competence Exam Success Jayson Zeigler University of Indianapolis FIELDWORK RELATIONSHIPS 2 Abstract A dilemma occurs when occupational therapy students earn perfect scores on level I fieldwork assessments, but fail future level II fieldwork experiences. The purpose of this study was to determine differences in level I fieldwork assessment scores, level II fieldwork assessment scores, Occupational Therapy Knowledge Exam (OTKE) competency scores, student demographics, and level I fieldwork factors in 21 Masters of Occupational Therapy (MOT) and 52 Occupational Therapy Doctorate (OTD) students from a private Midwestern university. Methods included parametric and non-parametric testing to determine these differences with no regression modeling warranted based on limited results. Results revealed statistically significant differences (p < .05) between two OTKE domains and tasks and one-week level I fieldwork assessment scores. Future research should address the psychometric development of level I fieldwork evaluation tools to ensure that these instruments are truly measuring clinical skills and professionalism criterion as intended. Implications of this study focus on expanding the scholarly agenda of the academic fieldwork coordinator to better understand the science of fieldwork education measurement, particularly during early experiential learning such as level I fieldwork. These actions then facilitate better identification for students at risk for level II fieldwork failure and validate better skill competence to provide high quality and value-based care by future occupational therapy practitioners. FIELDWORK RELATIONSHIPS 3 Table of Contents Title Page........................................................................................................................................1 Abstract...........................................................................................................................................2 Table of Contents............................................................................................................................3 Chapter 1: Introduction...................................................................................................................5 Background.........................................................................................................................5 Problem statement...............................................................................................................5 Purpose statement...............................................................................................................6 Research question...................................................................................................6 Study objectives......................................................................................................7 Significance of the study.....................................................................................................8 Chapter 2: Literature review...........................................................................................................8 Chapter 3: Method..........................................................................................................................16 Study design.......................................................................................................................16 Participants.........................................................................................................................16 Data....................................................................................................................................17 Operationalization and definitions.....................................................................................17 Instruments.........................................................................................................................18 Procedures..........................................................................................................................20 Participant identification........................................................................................20 Data extraction procedures....................................................................................20 Data management..................................................................................................22 FIELDWORK RELATIONSHIPS Data Analysis ........................................................................................................22 Chapter 4: Results..........................................................................................................................23 Chapter 5: Discussion....................................................................................................................25 Chapter 6: Conclusion....................................................................................................................40 References......................................................................................................................................43 Tables.............................................................................................................................................51 Table 1...............................................................................................................................51 Table 2...............................................................................................................................53 Table 3...............................................................................................................................54 Table 4...............................................................................................................................56 Appendix.......................................................................................................................................58 4 FIELDWORK RELATIONSHIPS 5 Determining the Relationship Between Level I Fieldwork Performance, Level II Fieldwork Performance, and Competence Exam Success Occupational therapy students demonstrate entry-level competence based on achievements in didactic coursework and fieldwork experiences. Many preventative actions are taken by education programs to help students construct learning skills and prepare for level II fieldwork (Cardell, Koski, Wahl, Rock, & Kirby, 2017; Giles, Carson, Breland, Coker-Bolt, & Bowman, 2017). Fieldwork education, level I followed by level II fieldwork, aids in the development of clinical skills and professional behaviors. In 2017, 9 (0.8%) occupational therapy doctorate (OTD) students and 319 (2.3%) masters of occupational therapy (MOT) students failed level II fieldwork (American Occupational Therapy Association [AOTA], 2018c). These statistics indicate that most occupational therapy students are demonstrating preparedness for entry-level work in the profession. While level I fieldwork is intended to prepare students for level II fieldwork, Johnson, Koenig, Verrier Pierson, Santalucia, and Wachter-Schutz (2006) highlighted that occupational therapy students perceive level I fieldwork (early experiential learning experiences) as not facilitating feelings of confidence and preparedness going into level II fieldwork. This further contributes to the dilemma of better understanding students who are at risk for level II fieldwork failure after completing level I fieldwork curricula in their academic programs. The dilemma further escalates when education programs determine students are failing level II fieldwork at a percentage above the national average, leading stakeholders to question what factors compromised student performance during the fieldwork education process. Problem Statement The significance of level I fieldwork student performance has been under researched and little is known about the relationship these early experiential learning experiences have with later FIELDWORK RELATIONSHIPS 6 level II fieldwork performance by occupational therapy students. Moreover, little is known about the relationship that level I student demographics, level I fieldwork factors, and competence exam success have with level I fieldwork performance and later level II fieldwork performance. It is required that occupational therapy students successfully complete and pass both fieldwork education levels (level I and level II) to complete degree requirements and to seek eligibility to take the National Board Certification for Occupational Therapy (NBCOT) examination (Accreditation Council for Occupational Therapy Education [ACOTE], 2012). In preparation for taking the NBCOT board exam, many programs use the Occupational Therapy Knowledge Exam (OTKE) as a preparatory action to gauge a students success in the academic program (NBCOT, 2018a). The OTKE was designed to assess knowledge and skills acquired in early experiential learning (level I fieldwork) and didactic coursework in advance of setting level II fieldwork goals (NBCOT, 2018a). This standardized competence exam tool has been shown to be effective in preparing the student to be successful on the NBCOT examination (Alexander, Perryman, & Rivers, 2015). These actions contribute to student and program success, but little is known about how performance in early experiential learning activities (level I fieldwork) prepares the students for future experiential learning activities (level II fieldwork) and for being successful in the academic program and on the NBCOT examination. Purpose Statement The purpose of this study was to investigate the relationship between occupational therapy student level I fieldwork performance and level II fieldwork performance, academic performance, and competence exam success. This study answered the following research question: Is there an association between level I fieldwork performance and level II fieldwork performance, academic performance, and competence exam success? FIELDWORK RELATIONSHIPS 7 Research objectives. To answer this question, the following objectives were addressed. 1. To examine if there is a difference in occupational therapy students level II fieldwork performance, as measured by performance assessment scores and pass/fail rates, when compared to their level I fieldwork performance (assessment score = 100/score less than 100). 2. To determine if there is a significant difference in occupational therapy student academic performance, as measured with occupational therapy grade point average (OT-GPA) at the time of graduation, compared to student level I fieldwork performance assessment (assessment score = 100/score less than 100). 3. To examine if there is a significant difference in occupational therapy student competence exam success, as measured with OTKE competence exam scores and percentage who scored above or below the national average, when compared to level I fieldwork performance (assessment score = 100/score less than100). 4. To determine if there is a significant difference in student demographics (gender, age, and degree type) and level I fieldwork factors (setting and site characteristics, supervision type, and discipline type of fieldwork educator) when compared to student level I fieldwork performance (assessment score = 100/score less than 100). 5. To determine predictors of level I fieldwork assessment scores based on statistically significant differences and correlations between level I fieldwork assessment scores (assessment score = 100/score less than 100), level II fieldwork assessment scores, academic performance, student demographics, level I fieldwork factors, and competence exam success. FIELDWORK RELATIONSHIPS 8 Significance of Study The significance of this study is that information gained from it can be used by occupational therapy educators and fieldwork coordinators to better understand how level I fieldwork experiences and factors, academic performance, and student demographics relate to level II fieldwork performance. This may allow them to identify students at-risk of not passing level II fieldwork. Educators and coordinators may also use study results to create a learning environment that will help students successfully transition from classroom to clinic, improve student outcomes in level II fieldwork performance, and prepare students for certification. Literature Review Occupational therapy literature has examined various components of fieldwork education. However, there is a paucity of literature that specifically investigates evaluation of fieldwork education. Roberts, Hooper, Wood, and King (2015) found 10% of occupational therapy fieldwork education literature focused on assessment and influence of specific fieldwork learning environments. Roberts et al. (2015) found fieldwork education literature mainly focuses on participants subjective perceptions of the fieldwork experience as outcomes versus examination of assessment data as outcomes. This literature review will explore the constructive process of fieldwork education, student characteristics related to fieldwork performance success, past predictors of fieldwork performance success, and the relationship between fieldwork performance and competence exam success. Fieldwork education is often described as the bridge from classroom to clinic. It assists occupational therapy students with making the transformation from the student role to the role of practitioner (Brzykcy, Geraci, Ortega, Tamra, & McWilliams, 2016). Most importantly, fieldwork facilitates the opportunity for students to develop performance-measured skills such as clinical FIELDWORK RELATIONSHIPS 9 reasoning, reflective practice, and professional knowledge on the normative beliefs of the field through professionalization while gaining clinical competence as a general practitioner (ACOTE, 2012; DeIuliis, 2017). Successful completion of both fieldwork education levels is needed for the student to be eligible to take the NBCOT examination (NBCOT, 2018b). With a goal of developing a generalist or general practitioner, fieldwork is organized by levels to include level I, an introductory-competence level and level II, a progressive-entry level (ACOTE, 2012). Level I fieldwork should reflect the curriculum design of the program and enhance coursework through observation and participation in the occupational therapy service delivery process (ACOTE, 2012). Sequence and duration of level I fieldwork experiences vary at the discretion of the occupational therapy program, but should be consistent with ACOTE standards (DeIuliis, 2017). Level II fieldwork; however, is more standardized across academic programs while at the same time being consistent with ACOTE standards (ACOTE, 2012). At each level of fieldwork, the occupational therapy student is assigned a fieldwork educator and exposed to various practice populations. The fieldwork educator observes and formally evaluates the students technical performance and professional behaviors. Roles of the fieldwork educator are not exclusively to be an evaluator, but also act as a site supervisor, educator facilitating professionalization, mentor, and clinical reasoning guide (Costa, 2015). Level I fieldwork is the first opportunity for students to participate in supervised clinical education that facilitates methods of constructivism to include transformational learning, experiential learning, situated learning, and reflective practice (DeIuliis, 2017; Giles et al., 2014; Merriam, Caffarella, & Baumgartner, 2007). Level I fieldwork focuses on the student internalizing and applying didactic knowledge, interpersonal characteristics and attributes, and technical (clinical) skills acquired throughout the curriculum in preparation for level II fieldwork (ACOTE, 2012; Giles et al., 2014). FIELDWORK RELATIONSHIPS 10 According to constructivism pedagogy, context of learning impacts learning itself while fieldwork facilitates an intentional opportunity to construct knowledge through interactions within a learning environment (Gredler, 1997; Merriam et al., 2007). Fieldwork education applies educational pedagogies, first described by John Dewey, based on experiential learning (DeIuliis, 2017). Experiential education enhances student learning by providing an environment in which they can construct interpersonal characteristics, attributes, and skills that may promote their development of professional competence (Cutchin, 2004; DeIuliis, 2017). Merriam et al. (2007) identified transformational, experiential, situated, and reflective practice learning as methods for implementing constructivism. Level I fieldwork provides the opportunity for the learner to construct a new sense of self where future self as practitioner is envisioned (Costa, 2015). It also creates a situation in which students participate in finding meaning and understanding of what is learned in the classroom, progress through the use of well-designed and appropriate clinical challenges, and think about and reflect on the experience (Dewey, 1938; Giles et al., 2014; Merriam et al., 2007). Roberts et al. (2015) stressed fieldwork research must have conceptual and theoretical underpinnings to link a study to much larger issues related to fieldwork within occupational therapy education. Past occupational therapy literature has emphasized the importance of constructivism as rationale for learning related to preparation for level II fieldwork, but specific aspects such as the constructive process of level I fieldwork have not been targeted in past studies to influence level II fieldwork. In 2017, 1,097 OTD students were placed for level II fieldwork, 1,088 placements were successfully completed resulting in a successful completion rate of 99% pass rate was reported (AOTA, 2018). For MOT students, 13,678 students placed for level II fieldwork, 13,359 placements were successfully completed, and a 98% pass rate reported (AOTA, 2018). Despite low FIELDWORK RELATIONSHIPS 11 failure rates, factors identifying at-risk students for failing level II fieldwork have not been established in the literature (Whisner, Geddie, Sechrist, & Wang, 2019). Preventing and understanding factors associated with level II fieldwork failure is difficult, especially if students succeed in their didactic coursework and pass level I fieldwork. Past studies examined fail rates within programs and focused on characteristics and attributes associated with successful and unsuccessful fieldwork students. Higher student academic achievement has been demonstrated to have a positive correlation with higher fieldwork performance scores (Howard & Jerosch-Herold, 2000; Tan, Meredith, & McKenna, 2004). Emotional intelligence and professional communication skills have also been correlated with higher fieldwork performance scores (Andonian, 2013; Andonian, 2017; Brown, Williams, & Etherington, 2016; DeIuliis, 2017; Tan et al., 2004; Tickle-Degnen, 1998). Supervision provided by fieldwork educators has been found to have the greatest influence on the development of attitudinal and interpersonal factors within students (Christie, Joyce, & Moeller, 1985; Hanson, 2011). DeIuliis (2017) listed several traits of successful fieldwork students ranging from strong self-awareness to strong organization, problem-solving, and communication skills. Negative attributes of unsuccessful fieldwork students range from lack of clinical competence to the inability to demonstrate appropriate safety and judgment (DeIuliis, 2017). These attributes are measured on level I and level II fieldwork performance evaluations and these outcomes are intended to be predictors of success for professional skill development (Cardell, Koski, Wahl, Rock, & Kirby, 2017; DeIuliis, 2017; Roberts et al., 2015). Brown, Caruso Streeter, Stoffel, and Mcpherson (1989) were the first to study how to measure occupational therapy student level I fieldwork performance. Using a level I fieldwork student performance evaluation tool they created, Brown et al. (1989) collected data from 259 occupational therapy and occupational therapy assistant students, fieldwork FIELDWORK RELATIONSHIPS 12 supervisors, and faculty. Their findings provided evidence that research efforts should focus on the value of level I fieldwork experiences and the development for criterion referenced level I fieldwork assessment tools (Brown et al., 1989). Brown et al. (1989) also generated the idea for questioning the ability of level I fieldwork assessment tools to provide critical feedback before level II fieldwork experiences. Koenig, Johnson, Morano, and Ducette (2003) validated and established the reliability of the Philadelphia Region Fieldwork Consortium Level I Student Evaluation (2018), a tool used to measure student professional behavior during level I fieldwork experiences. Their efforts suggested the establishment of psychometric properties for criterion such as professionalism in order to observe constructivism in student professional behaviors and identify at risk students early prior to participating in level II fieldwork (Koenig et al., 2003). The study found that students constructed professionalism skills within level I fieldwork experiences as evidenced by statistically significant differences found when comparing the professionalism level I fieldwork assessment scores from the first, second, to third experiences, p < .001 (Koenig et al., 2003). These findings suggest that when academic fieldwork coordinators try to psychometrically develop criterion on level I fieldwork assessment tools, the constructivism of skills such as professionalism can be observed during the assessment of the student and then discrimination among students can take place to provide remediation that will better socialize the student to the occupational therapy profession (Koenig et al., 2003). However, psychometric development in all criterion on level I fieldwork assessment tools is not yet a common and routine practice by academic fieldwork coordinators. Research presented in the literature facilitates awareness for qualities of successful and unsuccessful fieldwork students, but does not specifically focus on the value of the constructive process for developing these traits in level I fieldwork and potential relationship with and effects on level II fieldwork and professional competence development. FIELDWORK RELATIONSHIPS 13 Predictive factors associated with overall program performance and fieldwork performance have been studied and focused on predictive pre-admission and program completion factors. Preprogram grade point average (GPA), pre-program science GPA, and Graduate Record Examination (GRE) scores have been found to predict overall program performance, but not individual aspects of a program such as fieldwork performance. (Lysaght, Donnelly, & Villeneuve, 2009; Kirchner & Holm, 1997; Kirchner, Stone, & Holm, 2001; Whisner et al., 2019). Bathje, Ozelie, and Deavila (2014) revealed analytical and qualitative GRE scores do not predict fieldwork performance, but the scores from the written sub-scale of the GRE does have predictive qualities. Kirchner et al. (2001), in a previous study, revealed a positive correlation between analytical GRE scores and level II fieldwork performance scores. The literature reveals there are evidence-based non-predictors and predictors of level II fieldwork; however, prediction modeling for level I fieldwork students has not been studied. Investigating the relationship between level I fieldwork performance, level II fieldwork performance, and program success is valuable. Lastly, current literature has examined the relationship of competence exam success and fieldwork performance. Occupational therapy entry-level education relies on formal and summative tools such as the OTKE. These tools are used to determine if students are able to effectively utilize acquired clinical skills and knowledge gained through completion of didactic coursework and level I and level II fieldwork experiences (Avi-Itzhak & Krauss, 2010). The OTKE was designed to assess the acquired knowledge and skills thus far in didactic education and early experiential learning (level I fieldwork) and the results are used to set specific fieldwork goals in preparation for level II fieldwork (NBCOT, 2018a). The OTKE is 100 multiple-choice questions organized into four domains that align with the blueprint of the exam: (a) evaluation and assessment to acquire factors using a client-centered approach; (b) formulate conclusions based on analysis and FIELDWORK RELATIONSHIPS 14 interpretation of findings from assessment; (c) intervention management for client-centered care purposes; and (d) competency and practice management (NBCOT, 2018a). Each domain is then subdivided into task and knowledge statements derived from a large scale, validation practice analysis study that outlines essential competencies for entry-level practice. For example, Domain I, which involves responsibilities and duties relative to evaluation and assessment, is then subdivided into tasks that describe content knowledge and actions that describe acquisition and analysis of evidence when performing an occupational profile (NBCOT, 2018a). Domain II, pertaining to analysis and interpretation, is subdivided into two tasks that tests student knowledge on nonstandardized and standardized assessment findings and implementation of evidence-based practice for client-centered treatment (NBCOT, 2018a). Domain III, which includes intervention management, is subdivided into three tasks that examines student knowledge on their ability to manage interventions for the pediatric, young adult and adult, and geriatric populations within consideration for psychosocial, biomechanical, and neurological factors (NBCOT, 2018a). Domain IV, competency and practice management, is subdivided into activities that measure knowledge on life-long learning for competency development and ethical decision-making for safe practice for the consumer of occupational therapy services (NBCOT, 2018a). The blueprint of the NBCOT certification format is similar to the OTKE format as evidenced by the same domains to organize the exam, but with more tasks added in each domain testing knowledge of the student on the NBCOT certification exam (NBCOT, 2018b). In health professions, preparatory tools, like the OTKE, designed by board certification agencies, have been identified as a learning activity to help students gain familiarity with national certification exams. Preparatory tools like the OTKE also facilitate the opportunity to assess summative student knowledge and skill acquisition from clinical experiences and didactic FIELDWORK RELATIONSHIPS 15 coursework (Avi-Itzhak, 2015; Dadian et al., 2002; Edenfield & Hansen, 2000). In one study, researchers revealed standardized competence exam success (OTKE scores above national average) was not a predictor of level II fieldwork performance (Whisner et al., 2019). The literature highlighted that OTKE results have been found to effectively provide students with information on their strengths and weaknesses related to skill and knowledge acquisition, which is beneficial for cultivation of competent future, certified therapists (Alexander et al., 2015). Outcomes of the OTKE are beneficial both for the occupational therapy student and faculty. Results of the OTKE not only provide the student with performance data, but can also be used by occupational therapy faculty to analyze and enrich curriculum design, which includes didactic coursework and embedded level I fieldwork sequence (Breen-Franklin, 2017). The reviewed literature suggests early exposure to the expectations and format of the licensure examination process can benefit students by providing them with an assessment of their readiness to pass the NBCOT examination. For educators, results of the OTKE can be used to raise awareness for at-risk students (Breen-Franklin, 2017; Whisner et al., 2019). A gap in the literature is the failure to examine the relationship between early experiential learning experiences (level I fieldwork) and student competence exam success. Therefore, studying a potential association between level I fieldwork and student competence exam success (OTKE scores) is valuable to better understand the theoretical underpinnings of constructivism for learning within the occupational therapy education process. Giles et al. (2014) emphasized that an occupational therapy student in preparation for level II fieldwork demonstrates a responsibility for constructing their knowledge through their early experiences by participating in transformational, situated, and experiential learning while simultaneously developing reflective practice skills. Past literature has not gone past perception or descriptive studies to better understand the constructive process for professional behaviors, skills, FIELDWORK RELATIONSHIPS 16 and clinical reasoning development related to summative assessment data as outcomes (Roberts et al., 2015). In summary, this literature reveals much is unknown about the relationship between early level I fieldwork experiences, level II fieldwork, and later competence exam success. Understanding this relationship will provide insights into the use of level I fieldwork assessment data as an outcome measure for preparation for level II fieldwork, program performance, and competence exam success. Method Study Design This was a non-experimental study using a retrospective cohort design to determine if relationships exist between occupational therapy student level I fieldwork performance and level II fieldwork performance, academic performance, and competence exam success. The study took place from May 2019 January 2020. Prior to data extraction, the study was approved by the Universitys Human Research Protections Program (HRPP) and the Universitys Registrars Office Participants Program and institutional data for 73 entry-level MOT (n = 21) and OTD (n = 52) students (two different cohorts) who completed the program between 2014 and 2018 at the University of Indianapolis were extracted from student records. Participants included in the study were enrolled in the accredited occupational therapy program and routinely completed a one, two, and three-week level I fieldwork experiences that are sequenced in the academic programs curricula. The level I fieldwork experiences were full-time, off site, and within medical model settings (inpatient, schoolbased, home health, and outpatient settings). A fourth 8-week level I, community-based fieldwork FIELDWORK RELATIONSHIPS 17 experience was excluded from this study due to use of a different level I fieldwork performance evaluation tool. An a priori minimum sample size calculation was conducted using a formula specific for reliability studies, developed by Walter, Eliasziw, and Donner (1998). A reliability study sample size calculator was used assuming the study would be examining the correlation between level I and level II performance assessment scores. The following parameters were used for the sample size calculation;(a) significance level of .05; (b) power of .80; (c) acceptable reliability of .70; and (d) expected reliability of .50. Based on that calculation, a minimum sample size of 63 participants was recommended to sufficiently power this study. Therefore, 73 participants adequately powered this study. Data De-identified data were provided to the primary researcher (J. Z.) by the fieldwork administrative assistant. Data were extracted from four different sources: 1) CORE, the fieldwork data management system used by the OT program at the University; 2) Occupational Therapy Centralized Application Service (OT-CAS) used by the OT program at the University; 3) department Banner (learning management data system) portal; and 4) NBCOT portal. Data extracted from CORE included fieldwork performance scores (level I and level II) and fieldwork factor data. Fieldwork factor data included duration of fieldwork measured in hours, sequence of placement, setting and site characteristics, supervision type, and discipline. Demographic data included gender, age, race, and degree type extracted from the OT-CAS. Academic performance data included participant OT-GPA at time of graduation and was extracted with permission from the Registrars Office and in compliance with FERPA standards from department Banner portal access. The OTKE scores and OTKE completion date was extracted from the NBCOT portal. FIELDWORK RELATIONSHIPS 18 Operationalization and definitions. Academic performance was defined as cumulative OT-GPA at time of graduation. Supervision type was characterized by direct supervision by qualified personnel (1:1 model) or a collaborative model where level I peer groups were supervised by one qualified personnel (Costa, 2015). Site characteristics included duration of fieldwork measured by hours, sequence of placement documented as first (one week), second (two week), or third (three week), indication of full-time or part-time placement, setting and site characteristics, supervision type, and discipline type for fieldwork educator. For this study, a student will be categorized as passing level II fieldwork if a score of 122 or higher was obtained on the Level II Fieldwork Performance Evaluation or failing if the score was below 122 (AOTA, 2002). Instruments Level I fieldwork performance evaluation. At the end of each level I fieldwork experience, students are assessed using the University of Indianapolis Level I Fieldwork Performance Evaluation. This is a 33-item skill observation tool that evaluates an occupational therapy students task and professional behaviors while on a one, two, and three-week level I fieldwork evaluation. This tool was designed to measure skills and behaviors, planning, intervention, communication, critical reasoning, and professionalism. These items are measured on a 3-point scale with 3 = satisfactory, 2 = needs improvement, and 1 = unsatisfactory. In addition, if needed, the fieldwork educator may choose to select not applicable if a construct unable to be met in a fieldwork setting. Total scores range from 0-99 and are calculated and reported as a percentage. The University of Indianapolis tool does not define student ratings and scoring criteria. This assessment tool was designed using ACOTE objectives and program curriculum objectives (ACOTE, 2012). This tool has never been validated and examined for reliability. Fieldwork FIELDWORK RELATIONSHIPS 19 educators complete this assessment tool with the guidance of the instructions at top of the electronic form. Level II fieldwork performance evaluation. The AOTA Fieldwork Performance Evaluation (FWPE) is a 42-item assessment instrument designed to measure an occupational therapy students performance for applying the occupational therapy process within level II fieldwork (AOTA, 2012; DeIuliis, 2017). Each item is scored using a 4-point Likert-like scale with 1 = unsatisfactory, 2 = needs improvement, 3 = meets standards, to 4 = exceeds standards (AOTA, 2002). Rasch analysis methods revealed adequate range of items for level II fieldwork performance and acceptable standard error for each item, as well as 41 of the 42 items exhibiting acceptable goodness-to-fit and item separation (Atler, 2002; Bathje et al., 2014). Validity and reliability for this instrument is not established in the literature. Occupational therapy knowledge exam. The OTKE is a standardized online, 100-question examination that includes validated domains and tasks drawing on student clinical and didactic knowledge from accumulated coursework and experiential learning through fieldwork (NBCOT, 2018a). Psychometric establishment studies for this assessment are not released for public review and provided on a limited basis to occupational therapy academic program directors (NBCOT 2018a). Each year national averages are calculated and released to participating program chairs (NBCOT, 2018a). Students are provided results of their OTKE scores, which display raw scores, and percentages of the number of questions answered correctly in each validated domain and task, as well as their result in comparison to the national average. The OTKE was designed to assess the acquired knowledge and skills thus far in early experiential learning (level I fieldwork) to set specific fieldwork goals in preparation for level II fieldwork (NBCOT, 2018a). If below the FIELDWORK RELATIONSHIPS 20 national average, students within this program are required to develop a study plan for the NBCOT board exam in collaboration with their academic advisors at the university for which this study is taking place. At the time the studied cohorts were enrolled, the OT program at the University had students complete the OTKE after their first level II fieldwork rotation, which indicates dependence on skill and knowledge acquired and constructed during level I fieldwork is expected for success (above national average). Procedures Participant identification. Participants were identified from CORE fieldwork data management system by the fieldwork administrative assistant. The fieldwork administrative assistant created a list of potential participants in an Excel spreadsheet and will determine if each student meets study inclusion criteria. If a student was not eligible, the name was deleted from the Excel spreadsheet. Once it was determined all students meet study inclusion criteria, data extraction began. Data extraction procedures. The following procedures identifies the actions of the fieldwork administrative assistant and program chair completed in regards to the collection of identifiable data and the transfer of de-identified data to the primary researcher. CORE. The fieldwork administrative assistant extracted the following data and input the data into an Excel spreadsheet: level I fieldwork performance scores, level II fieldwork performance scores, and fieldwork factors. Occupational therapy centralized application service. The fieldwork administrative assistant and the program chair collected demographic data (gender, age, race, and degree type) from the OT-CAS portal. The OT-CAS portal is accessible through a password-protected account FIELDWORK RELATIONSHIPS 21 managed by the program chair. These data were added to the Excel spreadsheet by the fieldwork administrative assistant. National board for certification in occupational therapy portal. The administrative assistant and occupational therapy program chair accessed and transferred participant OTKE scores and percentages to the Excel spreadsheet to include raw score (number of questions answered correctly) and domain percentages (percentage of questions answered correctly in each domain). The portal is overseen by the program chair and is password protected. Banner. Banner is the University of Indianapolis registrar archive records data management system. The administrative assistant extracted participants OT-GPA and input it into the Excel spreadsheet. This concluded data extraction. The Excel spreadsheet with participant identifier (name) was maintained and managed by the fieldwork administrative assistant; the primary researcher did not have access to this file. The fieldwork administrative assistant was responsible for de-identifying the data and creating the Excel spreadsheet with the de-identified data. Transfer procedures. The following procedures were used to de-identify the data and transfer the newly created spreadsheet to primary researcher. 1. The administrative assistant assigned a unique study identification number to each participant and remove each participants name. The file was then saved using a new name. The Excel file that links participants names and student identification number remained with the administrative assistant and will be deleted at conclusion of study. 2. The newly created Excel file with the de-identified data was sent by the administrative assistant to the primary researcher via the Universitys email. FIELDWORK RELATIONSHIPS 3. 22 Once the primary researcher received the Excel file, data were categorized for each participant as passing or failing level II fieldwork based on the Level II Fieldwork Performance Evaluation score. The primary researcher determined whether the participants OTKE score was above or below national average and that data were added to the Excel spreadsheet. 4. The primary researcher then exported data from the Excel spreadsheet into IBM SPSS Statistics for Windows, Version 24.0 (IBM Corp., Armonk, NY) for data analysis. 5. The Excel spreadsheet maintained and managed by the primary researcher was deleted one month following conclusion of study. Data management. The primary researcher managed and stored the data in the Excel spreadsheet and SPSS file on a password protected department computer located in a locked in office. The administrative assistant stored the identifiable data Excel spreadsheet on a password protected department computer kept in a locked office. One month following conclusion of study, data collected during study for research purposes was deleted. Data Analysis All data were analyzed using IBM SPSS Statistics for Windows, Version 24.0 (IBM Corp., Armonk, NY). All tests were two-tailed and a significance level of less than .05 was considered statistically significant. Descriptive statistics were conducted on the entire sample. Normality of data was determined using the Shapiro-Wilk test. To determine a difference in level II fieldwork performance assessment scores between students who had a perfect score on the level I fieldwork performance assessment and those who did not have a perfect score on the assessment (objective 1), Fishers exact and Mann-Whitney U tests were conducted. To determine a difference in level II FIELDWORK RELATIONSHIPS 23 fieldwork pass/fail rates between students who had a perfect score on the level I fieldwork performance assessment and those who did not have a perfect score on the assessment, Fishers exact tests were conducted. To determine a difference in students OT-GPA and level I fieldwork assessment scores (objective 2), independent t tests were conducted. To determine if there were differences in level I fieldwork performance assessment scores (score < 100, score = 100) and OTKE scores (objective 3), as well as between students who score above or below the national average on the OTKE, independent t, Mann Whitney U, and Fishers exact tests were conducted. Fishers exact tests were also conducted to determine differences in level I fieldwork performance assessment scores by student demographics and by level I fieldwork factors (objective 4). The original proposal included conducting regression analysis to determine significant predictors of level I fieldwork performance assessment scores. However, due to lack of significance between level I and level II fieldwork performance assessment results, regression analysis was not warranted. Results Records from 73 students were included in the study. The majority of participants were under 26 years of age (n = 68, 93.2%), Caucasian, non-Hispanic (n = 67, 91.8%), and female (n = 68, 93.2%). Fifty-two (71.2%) participants were working on a doctorate degree while 21 (28.8%) were working on a masters degree in occupational therapy. All participants completed full-time fieldwork placements for all level I and level II fieldwork sequences. Descriptive statistics for grade point average upon graduation (OT-GPA) for participants were a mean (standard deviation) of 3.72 (0.15). Information on setting and site characteristics, supervision type, and fieldwork educator discipline type for each level I fieldwork placement in sequence (L1A, L1B, and L1C) can be found in Table 1. Descriptive statistics for level I and first level II (L2R1), and second level II (L2R2) FIELDWORK RELATIONSHIPS 24 Performance assessments can be found in Table 2 while competence exam scores (OTKE) and success rates are found in Table 3. Objective One: Level II Performance Scores by Level I Assessment Categories Non-parametric statistical tests were completed to determine differences in level II fieldwork performance assessment scores between level I fieldwork performance assessment score categories (score < 100, score = 100) for all three rotations. No statistically significant differences were found for any of the three level I fieldwork rotations as detailed in Table 2. For the second part of objective one, to determine if there was a significant difference in level II pass/fail rates by level I fieldwork performance assessment score categories, results from two-sided Fishers exact tests indicate there was not a significant difference in L2R1 and L2R2 pass rates by level I performance assessment score rotation sequence categories (see Table 2). Objective Two: Occupational Therapy Grade Point Average by Level I Assessment Categories OT-GPA was not statistically significantly different between participants who received and did not receive a perfect score on the first Level I fieldwork assessment, t(71) = -0.35, p = .726. There was also not a statistically significant difference in OT-GPA by Level I performance assessment score categories for the second and third assessments, t(71) = 0.99, p = .325; t(71) = 0.12, p = .904, respectively. Objective Three: Competence Exam Success by Level I Sequence Assessment Categories A statistically significant difference was found between OTKE domain two task one (OTKEd2t1) scores and L1A (Z = -2.02, p = .043) assessment scores, as well as OTKE domain four task one (OTKEd4t1) scores and L1A (Z = -2.25, p = .024) assessment scores. No statistically significant differences were found between participants OTKE scores in any other OTKE domains FIELDWORK RELATIONSHIPS 25 or tasks between participants who had perfect and non-perfect level I fieldwork assessments. In addition, results indicated there were no significant differences in OTKE scores above or below the national average between level I performance assessment score categories for any of the three level I assessments. All comparison results are shown in Table 3. Objective Four: Demographics and Level I Fieldwork Factors by Level I Assessment Categories Analyses were conducted to determine if there were statistically significant differences in participant demographics (gender, age, race, and degree type) and level I fieldwork factors (setting and site characteristics, supervision type, and fieldwork educators discipline type) compared to level I fieldwork assessment scores for all rotations. No statistically significant differences were found for any of the comparisons for all three level I fieldwork rotations as detailed in Table 4. Results of this study did not yield enough statistically significant differences to pursue regression modeling to determine predictors of level II fieldwork performance or competence exam success. Discussion Level I fieldwork is an important part of occupational therapy education. A common dilemma experienced by academic fieldwork coordinators is when occupational therapy students receive perfect scores on Level I fieldwork evaluations, and then fail future progressive level II fieldwork experiences. This is especially concerning since there are no variables offering insight into this digression within the fieldwork process. Therefore, the purpose of this study was to determine if there were significant differences between level I and level II fieldwork performance, academic performance, level I fieldwork factors, student demographics, and competence exam success in occupational therapy students. The only statistically significant differences identified were between FIELDWORK RELATIONSHIPS 26 one-week, level I fieldwork assessment scores, and overall competency development in students within specific knowledge domains and tasks outlined by the NBCOT (2018a) on the OTKE. The majority of results from this study revealed no major statistically significant differences or associations among most of the variables. Therefore, results of this study were not useful to determine any predictors of level II fieldwork performance. The current state of level I fieldwork research on predictors of future fieldwork performance reflects the absence of a valid and reliable assessment tools to measure fieldwork performance and the lack of adequate sample sizes to effectively power studies. Both accurate assessment tools and adequate sample size are needed to reveal if there is a relationship between level I and level II fieldwork performance and whether level 1 fieldwork performance and other factors can predict future fieldwork performance. This study, like other studies reviewed in this paper, did not use a tool that was discriminatory and had a small sample size of resulting in an underpowered study, making it impossible to make conclusions about the relationship between level I and level II fieldwork performance and other factors that may influence fieldwork performance. All patterns, principles, and relationships shown by the findings are detailed below. Statistically Significant Findings A statistically significant difference was found in one-week, level I fieldwork assessment scores, and domain two (formulate conclusions based on assessment findings) task one (acquisition of information during occupational profile) scores on the OTKE. Another significant difference was found between one week, level I fieldwork student assessment scores and domain 4, competence development for managing professional activities. Managing professional activities facilitate the acquisition of knowledge translation for evidence-based practice (OTKE domain four task one assessment scores) to promote quality care. The two significant findings from this study have to be FIELDWORK RELATIONSHIPS 27 interpreted with caution given the low statistical power of the study, the homogeneity of individual scores on level I and level II assessments scores and the OTKE scores, and the lack of a performance assessment tool in which the validity, reliability, and responsiveness has been established. Regarding specific domain and task scores related to significant findings, students who scored less than perfect scores on 1-week, level I fieldwork assessments exhibited higher median scores, 0.80 on competence development for domain two (formulate conclusions based on assessment findings) task one (acquisition of information during occupational profile) mean scores in comparison to students who had perfect one-week, level I assessment scores with their median score found to be 0.70. Further analysis revealed students who scored less than perfect scores on 1week, level I fieldwork assessments exhibited lower median scores, 0.50, on competence development for managing professional activities that facilitate the acquisition of knowledge translation for evidence-based practice (OTKE domain four task one assessment scores) in comparison to students who had perfect one-week, level I assessment scores with their median score found to be 0.67. Median scores related to the statistically significant differences found in these OTKE domains and tasks are important to note because the results provide an index of the average position of the OTKE distribution of scores for each finding (Portney & Watkins, 2009). These results further offer insight into unexpected central tendency identifying abnormal and inconsistent group (perfect or not perfect level I fieldwork assessment scores) patterns within OTKE domain and task scores. The results would not be assumed or expected if constructive, transformational, affective, and active engagement learning are taking place. The typical nature of the data assumed based on the distribution of scores if these pedagogies hold true would be those students who score less than 100 on the level I fieldwork assessment tool within different level I FIELDWORK RELATIONSHIPS 28 fieldwork experiences would have lower OTKE total, domain, and task scores related to acquisition of skill and competence for client-centered and evidence-based practice. Again, due to the limited sample size, lack of score variability, and lack of use of a valid and reliable assessment tool, these results cannot be confidently reported as true differences. Therefore, it cannot be stated with certainty that these occupational therapy fieldwork education pedagogies and philosophies are or are not taking place until more research is conducted. It has been posited the level I fieldwork implementation process closely resembles the level II fieldwork process regardless of program-specific sequence, duration, and setting type, as highlighted by Shalik (1990). The similarities between level I and level II fieldwork are further revealed by analysis of level I fieldwork data to show students are supervised daily as well as engaging in formal and informal evaluation of their performance across the continuum of occupational therapy fieldwork education (Shalik, 1990). Occupational therapy education, including fieldwork education, assumes the philosophy that constructivism is taking place in and beyond the classroom setting while actively engaging in clinical education (AOTA, 2018b). It is also true that students learn assumptions for their discipline in health professions and form knowledge of the profession as it is adopted and acquired in early experiential learning experiences (Hofer, 2006; Long, Mitchell, Chase, & Mineo, 2019). Costa (2015) stated student assumptions and beliefs are acquired, transformed, or not transformed in unidentified situations for topics such as evidencebased and client-centered practice in early experiential learning experiences such as level I fieldwork. Again, these findings need to be interpreted with caution until a larger sample size and valid and reliable performance assessment is established. FIELDWORK RELATIONSHIPS 29 Non-Significant Findings There were no statistically significant results for all other studied objectives in determining differences between level I fieldwork assessment scores (< 100, = 100), level II assessment scores, academic performance, demographics, and level I fieldwork factors. It is important to note two findings did approach the set alpha level of p < .05 when attempting to determine differences in two-week level I fieldwork assessment scores and OTKE domain three task three scores (intervention skills), 0.07, as well as determining difference in three-week level I fieldwork assessment scores and again OTKE domain four task one (managing professional activities that facilitate the acquisition of knowledge translation for evidence-based practice), 0.09. Previous studies found positive and negative relationships between higher academic achievement and level II fieldwork performance scores in students (Howard & Jerosch-Heorld, 2000; Tan et al., 2004). Howard and Jerosch-Herold (2000) found pre-admission qualifications to be poor predictors of student fieldwork and academic performance. Tan et al. (2004) found that academic achievement was a positive predictor of level II fieldwork performance in students. Therefore, all findings, including the findings of this study, are conflicting with no statistical differences found in OT-GPA and fieldwork performance, but have been found to have a relationship in one other study. This study is unique because it included level I fieldwork performance as a variable to determine if differences exist between academic performance. This study found no relationship between student demographics and fieldwork performance. Past studies did not report interpretation of results related to relationships between student demographics and fieldwork performance but focused more on relationships between pre-admission qualifications, academic performance, competence exam success, patient outcomes, and level II fieldwork performance (Kirchner & Holm, 1997; Kirchner, Stone, & Holm, 2001; Tan et al., 2004; FIELDWORK RELATIONSHIPS 30 Whisner et al., 2019). Again, adequate sample size in future studies are needed in order to determine if conflicting or identical results will be found. In regards to fieldwork factors, results of this study support findings reported from the Shalik (1990) level I fieldwork study that site characteristics and expectations vary so drastically that it is difficult to establish statistically significant differences in fieldwork performance (level I and level II) and level I fieldwork factors. Supervision type, fieldwork educator discipline, and hands-on participation during level I fieldwork have all been variables that were studied using qualitative designs and findings suggested relationship between level I fieldwork performance and level I fieldwork factors previously mentioned (Haynes, 2011; Heine & Bennett, 2009; Shalik 1990). However, the studied level I fieldwork assessment tool using a quantitative design was unable to detect differences between level I fieldwork assessment scores and level I fieldwork factors. Past studies have examined relationships and differences between pre-admission criteria, level II fieldwork performance, academic performance, level II fieldwork factors, student demographics, and competence exam success in occupational therapy students with limited results providing evidence these true differences and relationships exist. (Kirchner & Holm, 1997; Kirchner et al., 2001; Lysaght, Donnelly, & Villeneuve, 2009; Shalik, 1990; Whisner et al., 2019). Therefore, results of this study add to the very limited findings and suggest something is missing from level I fieldwork assessment tools and study design methods that is inhibiting genuine hypothesis testing with inferential statistics practices from occurring when attempting to study and examine the science on fieldwork education measurement of student performance. Findings of this study were not of a magnitude that would shift paradigms in current fieldwork education philosophies or pedagogies. However, the findings do suggest level I fieldwork assessment tools developed at the program level need to have better validity, reliability, and FIELDWORK RELATIONSHIPS 31 responsiveness for assessment criterion such as student skill acquisition and qualities of professionalism. Stakeholders interested in fieldwork education literature should consider the value of all of these non-significant outcomes. In other words, these findings may be pushing for more accountability by occupational therapy education programs to design and implement level I fieldwork assessment tools that are responsive in all clinical education settings through the establishment of psychometric properties. Despite the statistical outcome of this study, there is opportunity in future research to enhance the power of studies that focus on the science of fieldwork performance measurement in students. Limitations The limitations of this study are the lack of established fidelity of the level I and level II performance assessment tools, lack of variability in performance assessment scores, and a study that was too underpowered to detect student performance score differences between level I and level II fieldwork. It is conjectured the limitations of homogeneity of scores and lack of statistical power to detect differences was primarily caused by the use an assessment tool that did not have established validity and reliability. It is unknown whether the performance assessment tool actually measures student fieldwork performance, whether the tool is stable and has internal consistency to measure student performance, and importantly, whether the tool is responsive enough to detect small changes in performance scores. Also, the AOTA Fieldwork Performance Evaluation (2002) lacks psychometric evidence threatening validity and reliability. Despite the close resemblance of level I and level II fieldwork evaluation requirements and supervision processes of the student cited by Shalik (1990), no statistically significant differences were determined between small changes in student level I fieldwork assessment scores and pass/fail rates in level II fieldwork. FIELDWORK RELATIONSHIPS 32 The lack of variability in level I fieldwork assessment scores, subsequent level II fieldwork assessments scores, and OTKE scores may have been a direct result of the inability of assessments to measure and discriminate sensitive change in potential performance issues within specific criterion in any of the fieldwork I or II experiences. The majority of both OTD and MOT students had perfect scores on all level I fieldwork experiences. In fact, 92% of level I A, and 79% of level I B and C fieldwork students had perfect level I fieldwork assessments scores. It also important to note that limited variability was found in level II fieldwork assessment scores as evidenced by less than 2.7 % of students scoring less than 122 (failing score on level II fieldwork assessment tool) when compared to all level I fieldwork assessment score categories. If performance assessment tools were designed to more accurately measure student performance criteria and be more responsive to changes in student performance then there may be more variability in performance scores. More variability could not only increase the power for the study to detect differences between level I and level II scores, but also allow the use of more powerful statistical tests. Another limitation to this study is the participant population lacked diversity in gender, age, and race; however, students who participated in this study were representative of the student population in similarly sized occupational therapy education programs (AOTA, 2018c; Harvison, 2018). The sample population size (52 occupational therapy doctorate students, 21, masters of occupational therapy students) when compared to national maximums related to student-faculty ratios cited by AOTA (2018c), 64 for occupational therapy doctorate students and 60 for masters of occupational therapy students as class size in ratio to one faculty member at programs nationally. Future Research Addressing limitations of this study can be a starting point for future study considerations. First, future research efforts should focus on level I fieldwork assessment tool design and FIELDWORK RELATIONSHIPS 33 implementation within occupational therapy education program curricula. Steps should include examining the role of the academic fieldwork coordinator completing the task of managing fieldwork data in order to support the best student outcomes in fieldwork education at all levels. Examining this role of academic fieldwork coordinator will potentially find the extent to which program flexibility is needed for the development, actions needed to ensure measurement, responsiveness, and validation (fidelity criteria) of level I fieldwork assessments. It is unknown what facilitators and barriers exist for academic fieldwork coordinators to fully implement evidence-based practice in fieldwork education. Future research is recommended to understand the needed resources for academic fieldwork coordinators to establish psychometric proprieties for criterion on level I fieldwork assessments and to understand better how to eliminate barriers preventing such efforts. More studies are needed to understand better the value for designing evaluation mechanisms for level I fieldwork that are responsive and may reveal constructivism, transformative learning, and competency-based learning, which is an assumption and expectation of the current occupational therapy education philosophy in the classroom and in clinical education settings (AOTA, 2018b). Second, the evidence from this study and past studies have not been strong enough to reject assumptions that signature pedagogies (affective, transformative, and active engagement learning) are or are not taking place in level I fieldwork education (Kirchner & Holm, 1997; Kirchner, Stone, & Holm, 2001; Schaber, 2014; Tan et al., 2004; Whisner et al., 2019). This is due to statistical power lacking in these studies to effectively determine differences and relationships of level I fieldwork assessment scores, level II fieldwork assessment scores, academic performance, and competence exam success. Therefore, future studies to determine differences and relationships within fieldwork education measurement, academic performance, and competence exam success FIELDWORK RELATIONSHIPS 34 should consider adequate sample sizes, reasonable effect sizes, and strategically setting the alpha level of significance (Kellar & Kelvin, 2013: Portney & Watkins, 2009). These actions will increase the statistical power of future studies to better detect true differences in level I assessment scores, level II assessment scores, academic performance, competence exam success. Future studies should consider placing high value in determining statistical power of a study prior to implementing methods in order to avoid explaining results using theoretical models and theories that may have many interpretations in complex categories of occupational therapy education such as fieldwork education. Implications Literature supported implications based on the results of this study are limited. Costa (2015) and DeIuliis (2017) are the most current literature in occupational therapy education to provide examples of level I fieldwork assessment tools with general criterion set for professionalism (time management, interaction, clinical reasoning, verbal and nonverbal communication, participation in supervisory process, evaluation, and intervention skills). The scoring criteria for these sample level I fieldwork assessments are basic and generally range unsatisfactory to exceeds expectations with no subcategories for scoring criteria provided. An endorsed level I fieldwork assessment tool published by AOTA (2017), Level I fieldwork competency evaluation for OT and OTA students, possesses similar criterion and scoring criteria. However, there is no literature on the establishment of psychometric properties for level I fieldwork assessment tool criterion or scoring criteria at this time. The implications of this study are meant to empower academic fieldwork coordinators to become aware of this problem and focus on the development of fieldwork assessment tools to have better responsiveness. This will then facilitate the opportunity for the fieldwork educator to provide FIELDWORK RELATIONSHIPS 35 meaningful feedback to students in early experiential learning (level I fieldwork) that will facilitate the assumed constructivism and transformational learning pedagogies and philosophies. The journey continues for occupational therapy fieldwork education researchers to determine if true differences and relationships can be found in level I fieldwork assessment scores, level II fieldwork assessment scores, academic performance, and competence exam success in order to better inform educators of risk factors for student outcomes such as level II fieldwork failure. The primary lesson learned from this study is that the Level I fieldwork performance-based tool that was examined by the researcher lacks the ability to inform decision-making after detecting differences in student outcomes that would reveal performance skills or learner characteristics that are of concern or warrant remediation prior to level II fieldwork. It has also been found that level I fieldwork assessment tools need to be more responsive to detect differences in student fieldwork performance. Therefore, an agenda can be created to better understand if responsiveness in level I fieldwork assessment tools is the missing link to determine if a relationship or differences exist between level I fieldwork performance, level II fieldwork performance, academic performance, and competence exam success. It is advised if academic fieldwork coordinators feel confident their level I fieldwork assessment tools have established psychometric properties reliability measurement, responsiveness, and validation (fidelity criteria) confirmed, then multivariate analyses should be conducted (Mowbray, Holter, Teague, & Bybee, 2003). This action would hopefully lead to level I fieldwork assessment scores that have more variability due to enhanced detection of differences in student performance criterion such as clinical and professionalism skills. Specifically, exploratory factor analysis and confirmatory factor analysis should be conducted if warranted to ensure construct validity exists within level I fieldwork assessment tools. This would provide a starting point to FIELDWORK RELATIONSHIPS 36 better understand student competencies for clinical and professionalism skills to identify correlation between variables. This would then lead to the ability to perform regression modeling led by academic fieldwork coordinators to determine predictive relationship within each occupational therapy program. Whisner et al. (2019) are the most recent researchers to conduct regression analysis to determine if a predictive relationship existed between level II fieldwork performance, preadmissions criteria, academic performance, and competence exam success using structural equation modeling as a confirmatory factor analysis method. Their findings revealed relationships between student thinking type, pre-admissions GPA, and level II fieldwork (Whisner et al., 2019). However, due to the use of proxy variables for OTKE competence exam success the researchers highlighted a severe limitation to study results reducing statistical power for the variables studied (Whisner et al., 2019). Reasoning for reduced statistical power in this study is also explained by the limitation that the current standardized level II fieldwork assessment tool lacks psychometric evidence and thus also contributes to the low statistical power of this variable (Whisner, 2019). Yet, the lack of psychometric evidence should not be viewed by academic fieldwork coordinators as a barrier to implications focused on developing and studying fieldwork assessment tools. This study identifies areas of improvement for level I fieldwork measurement with actions to consider by academic fieldwork coordinators. The implication of academic fieldwork coordinators to utilize inferential statistics when reviewing level I fieldwork data on a more regular basis aligns with the Occupational Therapy Education Research Agenda (AOTA, 2018a) by promoting evidence-based practices for the creation of performance-based tools to measure student competency at all levels of fieldwork. Academic fieldwork coordinators should focus on the responsiveness of level I fieldwork assessment tools by examining the structure of these measures FIELDWORK RELATIONSHIPS 37 in order to consider adding subcategories for fieldwork educators to score critical variables related to skill acquisition in clinical environments (Kielhofner, 2006; Mowbray et al., 2003; Portney & Watkins, 2009). It is not enough for level I fieldwork assessments to have face validity, but rather time should be taken to ensure content, criterion, and construct validity is established in order to avoid the inability of the instrument to detect differences (Kielhofner, 2006; Portney & Watkins, 2009). An ideal level I fieldwork assessment would measure student performance constructs as it is intended, have precision of measurement, sensitivity, possess norm and criterion referencing, and have established methods for understanding standard error of measurement (Kielhofner, 2006). This would allow comparisons to be completed between students level I fieldwork assessment scores and depending less on the assumption that perfect level I fieldwork scores are indicating progression in all areas of skill acquisition as a student progress through an occupational therapy academic program. Then, the academic fieldwork coordinators need to educate fieldwork educators to ensure the competence for scoring student criterion and for the use of scored sub-categories that provide the opportunity for raters to highlight critically different variables of the student's performance (Mowbray et al., 2003). These methods facilitate validation and confirmation of student competence in certain areas of knowledge and skills measured on level I fieldwork assessments (Cook et al., 2015). The most recent literature focused on level I fieldwork assessment tool development and psychometric established is found in the works published by Koenig et al. (2003) and Brown et al. (1989). Both research teams started with a state consortium of academic fieldwork coordinators and fieldwork educators to identify needed criterion on level I fieldwork assessment tools. Koenig et al. (2003) focused on developing criterion for professionalism behaviors to be assessed during level I fieldwork experiences. Brown et al. (1989) focused their FIELDWORK RELATIONSHIPS 38 efforts of designing a tool that was comprehensive for a variety of level I fieldwork settings. Koenig et al. (2003) were able to provide evidence of constructivism for professionalism development in level I fieldwork students as evidenced by first rotation students scoring lower on their tool in comparison to scores on second and third level I fieldwork experiences (p < .001). However, both research teams provide a road map that can be added to the implication agenda suggested in this study for level I fieldwork assessment tool design. These stages of level I fieldwork assessment tool design recommended by Brown et al. (1989) and Koenig et al. (2003) are: stage one to focus on Level I fieldwork tool design with item analysis and factor analysis conducted after pilot implementation of tool in program decided level I fieldwork sequence; stage two to educate fieldwork educators and initiate of rater reliability studies to determine if assessment tool revision is needed; and stage three is to conduct validity studies to compare known groups such as the groups identified in this study as having perfect and not perfect level I fieldwork assessment scores. Another consideration when determining how to implicate the information from this study is to understand that level I fieldwork assessments must not be considered one size fits all for generations of students over time as a program evolves. Axioms drive this implementation mindset, and in this situation, the validation of differences for fieldwork performance in students is unidentifiable if level I fieldwork assessment tools are not frequently reassessed for quality improvement. DeIuliis (2017) has highlighted that generational differences are an intrinsic quality of a students professionalism perspective. Generational differences occur over time in occupational therapy student populations and academic program models for implementing level I fieldwork FIELDWORK RELATIONSHIPS 39 assessment tools should frequently practice quality improvement for fidelity criteria of these tools to validate differences in student performance (McGrew et al., 1994; Mowbray et al., 2003). This occurs again by academic fieldwork coordinators implementing multivariate analyses methods to understand better construct validity within criterion measured on level I fieldwork assessments. Academic fieldwork coordinators should then attempt to establish concurrent validity and predictive validity of all criterion on level I fieldwork assessment tools. This implication is necessary to better understand the relationship and determination of true differences between level I fieldwork performance, level II fieldwork performance, academic performance, and competence exam success because psychometric properties are established. In summary, implications based on information from this study can be used for academic fieldwork coordinators to form an agenda to participate better in establishing science for the measurement of level I fieldwork education. These implications include, but are not limited to: design program focused level I fieldwork assessment tool with consortium expertise from academic fieldwork coordinators, faculty, fieldwork educators, and consideration for literature supported student perception on level I fieldwork as cited by Johnson et al. (2006). educate fieldwork educators on level I fieldwork tool criterion; conduct exploratory and confirmatory factor analysis of clinical skills and professionalism criterion on level I fieldwork assessment tools; conduct rater reliability studies on design level I fieldwork assessment tool; consider revisions to the tool at this point with guidance from consortium efforts; establish concurrent validity and predictive validity for clinical skills (in the occupational therapy process) and professionalism criterion; FIELDWORK RELATIONSHIPS 40 conduct on-going reassessment of tool based on observation of level I fieldwork formed groups based on observed patterns in data; and repeat quality improvement actions cited above to ensure the reliability and validity of level I fieldwork measurement in your program in order to ensure constructivism, affective, and active engagement learning are truly taking place and reflected in fieldwork data. These implications may be interpreted as more ideas for future studies. However, StutzTanenbaum, Hanson, Koski, and Greene (2015) identified academic fieldwork coordinators as participating in the part-time role of data manager and using fieldwork data to better bridge didactic and clinical setting education. These implications are actions needed to ensure that future fieldwork data from existing databases do not have threats to reliability or validity due to level I fieldwork assessments tools having unknown psychometric properties or secondary analysis of the data is almost impossible due to the inability to properly operationalize variables on level I fieldwork assessment tools (Kielhofner, 2006). Therefore, these implications are providing an agenda for academic fieldwork coordinators to enhance their roles as data managers and be able to better promote the science for measurement of student fieldwork performance based on lessons learned from this study of a program developed level I fieldwork assessment tool. Conclusion Occupational therapy education in the past has assumed that level I fieldwork resembles level II fieldwork and this relationship allows for transformational and constructive learning to take place (Costa, 2015; Shalik, 1990). The level I fieldwork tool studied was unable to provide evidence of this relationship. However, this relationship is difficult to study due to the challenges associated with the lack of fieldwork assessment tools with established psychometric properties at FIELDWORK RELATIONSHIPS 41 both levels of fieldwork education. Health professions clinical education, in general, is being forced to focus on assessment challenges related to student performance due to recognized student deficiencies in critical competency areas such as professionalism, clinical evaluation and intervention skills, communication, and ethics (Mejicano, Klamen, Cate, Powell, & Lucey, 2017; Yoder-Wise, 2015). Bathe et al. (2014), as well as Roberts et al. (2015), recommended focusing on particular fieldwork experiences such as level I fieldwork experiences and indicators for level II fieldwork failures, which is achieved in this study. Level I fieldwork is further challenging to study due to particular problems and local conditions that limit generalizations and theory building due to level I fieldwork curricula not being standardized or existing data for secondary analysis posing threats to reliability and validity. However, there is still a need to address such fieldwork experiences through the implementation of theory building that links research questions to data and facilitates linkage to unfavorable outcomes such as level II fieldwork failure that can be resolved utilizing data-driven decision making (Berliner, 2002). Using a quantitative approach, this study also supplements scarce fieldwork education literature in specific research agenda categories such as theory building and pedagogy by examining theories of constructivism, transformational learning, and active engagement learning which are all underlying assumptions of occupational therapy fieldwork education (AOTA, 2018a; Zeigler et al., 2019). Moreover, this study aligns with the current Education Research Agenda (AOTA, 2018a) by promoting evidence-based practices for the creation of performance-based tools to measure student competency at all levels of fieldwork. The results of this study reveal value in both the minimal significant and non-significant findings by bringing awareness for the need for academic fieldwork coordinators as data managers to focus on the fidelity of measurement for level I fieldwork and consider an agenda for level I fieldwork assessment tool design. In conclusion, this study FIELDWORK RELATIONSHIPS 42 demonstrates the need to understand concepts for level I fieldwork assessment of student performance in early experiential learning that will ensure that occupational therapy fieldwork education at all levels is preparing students for board certification appropriately in all competency areas and to provide high-value medical care to society. FIELDWORK RELATIONSHIPS 43 References Accreditation Council for Occupational Therapy Education. (2012). 2011 Accreditation council for occupational therapy education (ACOTE) standards. American Journal of Occupational Therapy, 66(Suppl. 1), 1-68. Alexander, A., Perryman, M., & Rivers, P. A. (2015). Development and effectiveness of a national board certification for occupational therapists (NBCOT) examination preparatory course. Journal of Healthcare Finance, 42(2), 1-12. Atler, K. (2002). The complete guide: Using the fieldwork performance evaluation forms. State College, PA: AOTA Press. American Occupational Therapy Association. (2002). Fieldwork performance evaluation for the occupational therapy student. Bethesda, MA: AOTA Press. American Occupational Therapy Association. (2017). Level I fieldwork competency evaluation for OT and OTA students. Retrieved from https://www.aota.org/~/media/Corporate/Files/EducationCareers/Educators/Fieldwork/Level I/Level-I- Fieldwork-Competency-Evaluation-for-ot-and-ota-students.pdf American Occupational Therapy Association. (2018a). Occupational therapy education research agenda- revised. American Journal of Occupational Therapy, 72(2), 1-5. American Occupational Therapy Association. (2018b). Vision 2025. Information retrieved from https://www.aota.org/publications-news/otp/archive/2018/vision-2025.aspx American Occupational Therapy Association. (2018c). Academic programs annual data report: Academic year 2017 2018. Information retrieved from https://www.aota.org/~/media/Corporate/Files/EducationCareers/Educators/2017-2018Annual-Data-Report.pdf FIELDWORK RELATIONSHIPS 44 Andonian, L. (2013). Emotional intelligence, self-efficacy, and occupational therapy students fieldwork performance. Occupational Therapy in Health Care, 27, 1-15. https://doi.org/10.3109/07380577.2012.763199 Andonian, L. (2017). Emotional intelligence: An opportunity for occupational therapy. Occupational Therapy in Mental Health, 33(4), 1-9. https://doi.org/10.1080/0164212X.2017.1328649 Avi-Itzhak, T. (2015). Ability of the national board for certification in occupational therapy practice test to estimate the probability of first-time pass status on the national certification exam. American Journal of Occupational Therapy, 69, 1-8. doi:10.5014/ajot.2015.015024 Avi-Itzhak, T., & Krauss, A. (2010). An outcome assessment for evaluating occupational therapy students' proficiency and skills required for passing the NBCOT computer-based certification test. Journal of Allied Health, 39(2), 1-7. Bathje, M., Ozeilie, R., & Deavila, E. (2014). The relationship between admission criteria and fieldwork performance in a masters-level OT program: Implications for admissions. The Open Journal of Occupational Therapy, 2(3), 1-6. https://doi.org/10.15453/2168-6408.1110 Berliner, D. C. (2002). Comment: Education research. The hardest science of all. Educational Researcher, 31(8), 1-2. doi:10.3102/0013189X031008018 Breen-Franklin, A. (2017). Enabling student success for the NBCOT exam: A pilot education program. Occupational Therapy in Health Care, 31(4), 1-7. doi:10.1080/07380577.2017.1376766 Brown, S., Caruso Streeter, L. A., Stoffel, V. C., & McPherson, J. J. (1989). Development of a level I fieldwork evaluation. American Journal of Occupational Therapy, 43(10), 1-5. FIELDWORK RELATIONSHIPS 45 Brown, T., Williams, B., & Etherington, J. (2016). Emotional intelligence and personality traits as predictors of occupational therapy students practice education performance: A crosssectional study. Occupational Therapy International, 23, 1-13. https://doi.org/10.1002/oti.1443 Brzykcy, D., Geraci, J., Ortega, R., Tamra, T., & McWilliams, K. (2016). Occupational therapy fieldwork education: Value and purpose. American Journal of Occupational Therapy, 70(Suppl 2), 1-2. https://doi.org/10.5014/ajot.63.6.821 Cardell, B., Koski, J., Wahl, J., Rock, W., & Kirby, A. (2017). Underperforming students: Factors and decision-making in occupational therapy programs. Journal of Occupational Therapy Education, 1(3), 1-13. doi:10.26681/jote.2017. 010301 Christie, B., Joyce, P., & Moeller, P. (1985). Fieldwork experience, part II: The supervisor's dilemma. American Journal of Occupational Therapy, 39(10), 1-7. Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kanes framework. Medical Education, 49, 1-15. Costa, D. M. (2015). The essential guide to occupational therapy fieldwork education: Resources for educators and practitioners. Bethesda, MD: AOTA Press. Cutchin, M. (2004). Using Deweyan philosophy to rename and reframe adaptation-toenvironment. American Journal of Occupational Therapy, 58(3), 1-10. Dadian, T., Guerink, K., Olney, C., & Littlefield, J. (2002). The effectiveness of a mock board experience in coaching students for the dental hygiene national board examination. Journal of Dental Education, 66(5), 1-6. DeIuliis, E. D. (2017). Professionalism across occupational therapy practice. Thorofare, NJ: SLACK Incorporated. FIELDWORK RELATIONSHIPS 46 Dewey, J. (1938). Experience and education. New York, NY: Simon & Schuster Edenfield, S. M., & Hansen, J. R. (2000). Relationships among dental hygiene course grades, a mock board dental hygiene examination, and the National Board Dental Hygiene Examination. The Journal of Dental Hygiene, 72(2), 1-6. Field, A. (2013). Discovering statistics using IBM SPSS statistics (4th ed.). Thousand Oaks, CA: SAGE publications. Giles, A. K., Carson, N. E., Breland, H. L., Coker-Bolt, P., & Bowman, P. J. (2014). Use of simulated patients and reflective video analysis to assess occupational therapy students preparedness for fieldwork. American Journal of Occupational Therapy, 68, 1-10. http://dx.doi.org/10.5014/ajot.2014.685S03 Gredler, M. E. (1997). Learning and instruction: Theory into practice (3rd ed.). Upper Saddle River, NJ: Prentice-Hall. Hanson, D. (2011). The perspectives of fieldwork educators regarding level II fieldwork students. Occupational Therapy in Health Care, 25(2-3), 1-14. Harvison, N. (2018). Current trends in accreditation and higher education [PowerPoint Slide Presentation]. Retrieved from https://www.aota.org/~/media/Corporate/Files/EducationCareers/2018-conferencepresentations/2018-Trends-in-Accreditation-Higher-Education.pdf Haynes, C. J. (2011). Active participation in fieldwork level I: Fieldwork educator and student perceptions. Occupational Therapy in Health Care, 25(4), 1-13. https://doi.org/10.3109/07380577.2011.595477 Heine, D., & Bennett, N. (2009). Student perceptions of level I fieldwork supervision. Occupational Therapy in Health Care, 17(2), 1-9. https://doi.org/10.1080/J003v17n02_06 FIELDWORK RELATIONSHIPS 47 Hofer, B. K. (2006). Domain specificity of personal epistemology: Resolved questions, persistent issues, and new models. International Journal of Educational Research, 45(1-2), 85-95. https://doi.org/10.1016/j.ijer.2006.08.006 Howard, L., & Jerosch-Herold, C. (2000). Can entry qualifications be used to predict fieldwork and academic outcomes in occupational therapy and physiotherapy students? British Journal of Occupational Therapy, 63(7), 1-6. https://doi.org/10.1177%2F030802260006300706 Johnson, C. R., Koenig, K. P., Verrier Piersol, C. Santalucia, S. E., & Wachter-Schutz, W. (2006). Level I fieldwork today: A study of contexts and perceptions. American Journal of Occupational Therapy, 60, 1-13. Kellar, S. P., & Kelvin, E. A. Munros statistical methods for health care research (6th ed.). Philadelphia, PA: Wolters Kluwer Health, Lippincott, Williams, & Wilkins. Kielhofner, G. (2006). Research in occupational therapy: Methods of inquiry for enhancing practice. Philadelphia, PA: F. A. Davis Company. Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entry-level masters program. American Journal of Occupational Therapy, 51(9), 1-5. https://doi.org:10.5014/ajot.51.9.775 Kirchner, G. L., Stone, R.G., & Holm, M.B. (2001) Use of admission criteria to predict performance of students in an entry-level masters program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), 1-10. https://doi.org/10.1080/J003v13n01_01 Koenig, K., Johnson, C., Morano, C. K., & Ducette, J. P. (2003). Development and validation of a professional behavior assessment. Journal of Allied Health, 32(2), 1-6. FIELDWORK RELATIONSHIPS 48 Long, D. M., Mitchell, A. W., Chase, C., & Mineo, B. (2019). Entry-level occupational therapy students beliefs about knowledge and knowing: Findings from three masters level programs in the US. Journal of Occupational Therapy Education, 3(4), 1-26. https://doi.org/10.26681/jote.2019.030405 Lysaght, R., Donnelly, C., & Villeneuve, M. (2009). Factors predicting applicant outcomes in occupational therapy education. Canadian Journal of Occupational Therapy, 76(1), 1-10. https://doi.org/10.1177%2F000841740907600110 McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of mental health program. Journal of Consulting and Clinical Psychology, 62, 1-8. Mejicano, G., Klamen, D., Cate, O. T., Powell, D., & Lucey, C. (2017, June). Achieving competency-based, time-variable health professions education. Paper presented at Josiah Macy Junior Foundation: Achieving Competency-Based, Time-Variable Health Professions Education Conference. New York, NY. Retrieved from https://macyfoundation.org/assets/reports/publications/jmf2017_annual_report_webpdf.pdf Merriam, S. B., Caffarella, R. S., & Baumgartner, L. M. (2007). Learning in adulthood: A comprehensive guide. San Francisco, CA: Wiley. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 1-25. National Board for Certification in Occupational Therapy. (2018a). Exam preparation. Retrieved from https://www.nbcot.org/Educators/Exam-Prep FIELDWORK RELATIONSHIPS 49 National Board for Certification in Occupational Therapy. (2018b). NBCOT study guide for the OTR certification examination. Gaithersburg, MD: National Board for Certification in Occupational Therapy, Inc. Philadelphia Region Fieldwork Consortium. (2018). Philadelphia Region Fieldwork Consortium Level I Student Evaluation (2nd ed.). Retrieved from http://www.philaotfwconsortium.org/resources Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Application to practice. Upper Saddle River, NJ: Pearson Education, Inc. Roberts, M. E., Hooper, B. R., Wood, W. H., & King, R. M. (2015). An international systematic mapping review of fieldwork education in occupational therapy. Canadian Journal of Occupational Therapy, 82(2), 1-13. doi:10.1177/0008417414552187 Schaber, P. (2014). Conference Proceedings - Keynote address: Searching for and identifying signature pedagogies in occupational therapy education. American Journal of Occupational Therapy, 68, 1-5. http:dx.doi.org/10.5014/ajot.2014.685S08 Shalik, L. D. (1990). The level I fieldwork process. American Journal of Occupational Therapy, 44(8), 1-7. Stutz-Tanenbaum, P., Hanson, D., Koski, J., & Greene, D. (2015). Exploring the complexity of the academic fieldwork coordinator role. Occupational Therapy in Health Care, 29(2), 139-52. doi:10.3109/07380577.2015.1017897 Tan, K., Meredith, P., & McKenna, K. (2004). Predictors of occupational therapy students clinical performance: An exploratory study. Australian Occupational Therapy Journal, 51(1), 1-9. https://doi.org/10.1046/j.1440-1630.2003.00383.x FIELDWORK RELATIONSHIPS 50 Tickle-Degnen, L. (1998). Working well with others: The prediction of students clinical performance. American Journal of Occupational Therapy, 52(2), 1-10. https://doi.org/10.5014/ajot.52.2.133 Walter, S., Eliasziw, M., & Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine, 17(1), 1-11. Whisner, S. M., Geddie, M., Sechrist, D., & Wang, E. (2019). Examination of potential factors to predict fieldwork performance: A program evaluation project. Journal of Occupational Therapy Education, 3(1), 1-22. https://doi.org/10.26681/jote.2019.030106 Yoder-Wise, P. (2015). The continuously learning healthy system: Recommendations from the Josiah Macy Jr. foundation. Journal of Continuing Education in Nursing, 48(9), 1-2. doi:10.3928/00220124-20150821- 10 Zeigler, J., Janes, W., Cook, A., Escher, A., Cohill, B., Galey, S., . . . Olson, D. (2019, October). A mapping review of occupational therapy fieldwork literature. Paper presented at the meeting of American Occupational Therapy Association Education Summit 2019 Las Vegas, NV. FIELDWORK RELATIONSHIPS 51 Table 1 Descriptive Site Characteristics of Level I Fieldwork (N = 73) L1A, 1-week duration or 40 hours N (%) Setting and Site Characteristics Physical Dysfunction Setting 8 (11.0) Other 65 (89.0) Supervision Type 1:1 Other 72 (98.6) 1 (1.4) Fieldwork Educator Discipline Type OTR 47 (64.4) COTA or other 26 (35.6) L1B, 2-week duration or 80 hours N (%) Setting and Site Characteristics Physical Dysfunction Setting Other 7 (9.6) 66 (90.4) Supervision Type 1:1 Other 70 (95.9) 3 (4.1) Fieldwork Educator Discipline Type OTR 43 (58.9) COTA or other 30 (41.1) L1C, 3-week duration or 120 hours N (%) FIELDWORK RELATIONSHIPS 52 Setting and Site Characteristics Physical Dysfunction Setting Other 6 (8.2) 67 (91.8) Supervision Type 1:1 Other 71 (97.3) 2 (2.7) Fieldwork Educator Discipline Type OTR 37 (50.7) COTA or other 36 (49.3) Note. L1A = Level 1A fieldwork; L1B = Level 1B fieldwork; L1C = Level 1C fieldwork; OTR = Occupational Therapist; COTA = Certified Occupational Therapy Assistant FIELDWORK RELATIONSHIPS 53 Table 2 Comparison of Level II Performance Scores by Level I Performance Assessment Score Categories (N = 73) L1A Score L2R1 Score L2R2 Score 100 < 100 100 < 100 100 N=6 N = 67 N = 15 N = 58 N = 15 N = 58 M (SD) M (SD) p M (SD) M (SD) p M (SD) M (SD) p 138.33 133.97 .244 131.07 135.17 .265 134.73 134.22 .859 (9.99) (11.76) (14.90) (10.61) (9.11) (12.26) 136.50 138.13 129.33 140.24 138.60 137.84 (7.94) (20.82) (36.46) (12.50) (7.98) (22.17) N (%) N (%) N (%) N (%) N (%) N (%) .445 p 1.000 2 (2.7) 2 (2.7) L2R2 Score 2 (2.7) > 122 71 (97.3) 71 (97.3) p 2 (2.7) 2 (2.7) 71 (97.3) 71 (97.3) p 2 (2.7) 71 (97.3) 71 (97.3) .371 2 (2.7) .929 1.000 2 (2.7) 71 (97.3) 71 (97.3) 1.000 2 (2.7) .469 .371 2 (2.7) > 122 71 (97.3) 71 (97.3) < 122 L1C Score < 100 L2R1 Score < 122 L1B Score 1.000 2 (2.7) 2 (2.7) 71 (97.3) 71 (97.3) Note. L1A = Level 1A fieldwork; L1B = Level 1B fieldwork; L1C = Level 1C fieldwork; L2R1 = First Level II fieldwork placement; L2R2 = Second Level II fieldwork placement. FIELDWORK RELATIONSHIPS 54 Table 3 Comparison of OTKE Scores by Level I Performance Assessment Score Categories (N = 73) L1A Score L1B Score L1C Score < 100 100 < 100 100 < 100 100 N=6 N = 67 N = 15 N = 58 N = 15 N = 58 M (SD) M (SD) p M (SD) M (SD) p M (SD) M (SD) p Total OTKE 62.83 61.33 .565 61.47 61.45 .992 61.87 61.34 .770 Score (6.43) (6.09) (5.47) (6.28) (4.85) (6.40) Mdn Mdn Mdn Mdn Mdn Mdn (IQR) (IQR) (IQR) (IQR) (IQR) (IQR) 0.83 0.78 0.78 0.78 0.78 0.78 (0.13) (0.13) (0.33) (0.22) (0.11) (0.22) 0.63 0.63 0.63 0.63 0.63 0.63 (0.25) (.25) (0.13) (0.25) (0.25) (0.25) 0.80 0.70 0.70 0.70 0.70 0.70 (0.10) (0.20) (0.20) (0.20) (0) (0.20) 0.61 0.61 0.61 0.61 0.61 0.61 (0.06) (0.16) (0.11) (0.11) (0.16) (0.08) 0.66 0.56 0.55 0.57 0.56 0.57 (0.36) (0.19) (0.14) a (0.11) a (0.13) a (0.11) a 0.59 0.55 0.59 0.54 0.59 0.55 (0.13) a (0.11) a (0.09) a (0.11) a (0.14) (0.14) OTKEd1t1 OTKEd1t2 OTKEd2t1 OTKEd2t2 OTKEd3t1 OTKEd3t2 p .117 .708 .043* .934 .604 .497 p .989 .615 .989 .270 .808 .151 p .719 .571 .878 .807 .847 .461 FIELDWORK RELATIONSHIPS OTKEd3t3 OTKEd4t1 OTKEd4t2 0.43 0.43 (0.11) (0.14) 0.50 0.67 (0.34) (0.33) 0.71 0.57 (0.18) (0.14) 55 .303 .024* .728 0.43 0.43 (0.28) (0.14) 0.67 0.67 (0.33) (0.33) 0.71 0.57 (0.14) (0.14) .074 .723 .279 0.43 0.43 (0.14) (0.14) 0.67 0.67 (0.34) (0.33) 0.57 0.71 (0.28) (0.14) .668 .099 .774 Note. OTKE = Occupational Therapy Knowledge Exam. OTKEd1t1 = OTKE domain 1 task 1 score. OTKEd1t2 = OTKE domain 1 task 2 score. OTKEd2t1 = OTKE domain 2 task 1 score. OTKEd2t2 = OTKE domain 2 task 2 score. OTKEd3t1 = OTKE domain 3 task 1 score. OTKEd3t2 = OTKE domain 3 task 2 score. OTKEd3t3 = OTKE domain 3 task 3 score. OTKEd4t1 = OTKE domain 4 task 1 score. OTKEd4t2 = OTKE domain 4 task 2 score. L1A = Level 1A fieldwork. L1B = Level 1B fieldwork. L1C = Level 1C fieldwork. In order of level I fieldwork sequence for participants program reported in this study. aMean (standard deviation) reported. *p < .05. FIELDWORK RELATIONSHIPS 56 Table 4 Comparison between Level I Fieldwork Student Demographics and Site Characteristics by Level I Performance Assessment Categories (N = 73) L1A Score L1B Score <100 100 < 100 100 < 100 100 N=6 N = 67 N = 15 N = 58 N = 15 N = 58 N (%) N (%) N (%) N (%) N (%) N (%) p 1.000 Race White, NH Other L1C Score p .097 .335 6 (8.2) 61 (83.6) 12 (16.4) 55 (75.3) 15 (20.5) 52 (71.2) 0 (0) 6 (8.2) 3 (4.1) 3 (4.1) 0(0) 6 (8.2) Gender 1.000 .576 .576 Female 6 (8.2) 62 (84.9) 15 (20.5) 53 (72.6) 15 (20.5) 53 (72.6) Male 0 (0.0) 5 (6.8) 0 (0) 5 (6.8) 0 (0) 5 (6.8) Age 1.000 1.000 0 (0.0) 5 (6.8) 1 (1.4) 4 (5.5) 1 (1.4) 4 (5.5) < 26 6 (8.2) 62 (84.9) 14 (19.2) 54 (74.0) 14 (19.2) 54 (74.0) 1.000 1.000 .204 MOT 2 (2.7) 19 (90.5) 4 (5.5) 17 (23.3) 2 (2.7) 19 (26.0) OTD 4 (5.5) 48 (65.8) 11 (15.1) 41 (70.7) 13 (17.8) 39 (53.4) Setting/Site 1.000 .332 .596 Characteristics PD Setting Other 6 1.000 > 26 Degree Type p 0 (0) 8 (11.0) 0 (0.0) 7 (9.6) 2 (2.7) 4 (5.5) 6 (8.2) 59 (90.8) 15 (20.5) 51 (77.3) 13 (17.8) 54 (80.6) FIELDWORK RELATIONSHIPS Supervision 57 1.000 1.000 1.000 Type 1:1 Other 6 (8.2) 66 (90.4) 15 (20.5) 55 (75.3) 15 (20.5) 56 (76.7) 0 (0) 1 (1.4) 0 (0) 3 (4.1) 0 (0) 2 (2.7) Disciple of FE .412 1.000 .247 OTR 5 (6.8) 42 (89.4) 9 (12.3) 34 (46.6) 10 (13.7) 27 (37.0) COTA or Other 1 (1.4) 25 (34.2) 6 (8.2) 24 (32.9) 5 (6.8) 31 (42.5) Note. L1A = Level 1A fieldwork; L1B = Level 1B fieldwork; L1C = Level 1C fieldwork; NH = Non-Hispanic; OTR = Occupational Therapist Registered; COTA = Certified Occupational Therapy Assistant; FE = Fieldwork Educator; MOT = Masters of Occupational Therapy degree; OTD: Occupational Therapy Doctorate degree; PD = Physical dysfunction setting. FIELDWORK RELATIONSHIPS 58 Appendix University of Indianapolis Level I Fieldwork Performance Assessment Tool University of Indianapolis Student: Fieldwork Educator: Site: Date: () Level I Fieldwork Student Evaluation Form (FWPE) General overview Please respond to items 1-26 by selecting satisfactory (S), needs improvement (NI), or unsatisfactory (U) columns. Designate N/A if the item is not applicable. Please add clarifying statements and /or examples in the current column. Students must achieve a total score 80% to pass with no more than 1 item rated as Unsatisfactory. Comments are required for needs improvement and unsatisfactory ratings ASSESSMENT Identifies deficits in areas of occupation, occupational performance skills and patterns, and client factors across the lifespan. NO SCORE SELECTED Question Comments : Describes the impact of activity demands, environment, and context on occupational performance. NO SCORE SELECTED Question Comments : Completes an occupational profile on a client. NO SCORE SELECTED Question Comments : Consider factors that might bias assessment results. Question Comments : NO SCORE SELECTED FIELDWORK RELATIONSHIPS Compare and contrast the role of the OTA in the screening and evaluation process at this practice setting. 59 NO SCORE SELECTED Question Comments : Describes the impact of activity demands, environment, and context on occupational performance NO SCORE SELECTED Question Comments : PLANNING Interpret the results of the OT assessment with application made to the impact of the results on occupational performance. NO SCORE SELECTED Question Comments : Establishes at least two relevant and attainable goals, which are client centered and occupation based. NO SCORE SELECTED Question Comments : Describes services occupational therapy can offer patients/clients within this practice setting. NO SCORE SELECTED Question Comments : INTERVENTION Chooses or designs at least one occupation based intervention that is relevant to client needs. NO SCORE SELECTED Question Comments : Articulates the rationale for this intervention NO SCORE SELECTED FIELDWORK RELATIONSHIPS 60 Question Comments : Adheres to safety precautions and infection control guidelines during participation in all observed interventions NO SCORE SELECTED Question Comments : Articulates the purpose of observed therapist interventions to either staff or client. NO SCORE SELECTED Question Comments : Identifies and demonstrates understanding of the role of the OTA at this setting. NO SCORE SELECTED Question Comments : COMMUNICATION Uses effective strategies to interact and collaborate with staff, clients, and significant others, including the OTA. (ie. Relevant conversation, timing of questions, maintaining role of student). NO SCORE SELECTED Question Comments : Uses effective non-verbal communication with client and significant others (ie.body language, eye contact) NO SCORE SELECTED Question Comments : Completes assigned written documentation with supervision Question Comments : NO SCORE SELECTED FIELDWORK RELATIONSHIPS 61 CRITICAL REASONING Describes the various aspects of the OT process in this practice setting. NO SCORE SELECTED Question Comments : Applies through written assignments and/or conversations with FW educator and/or client, classroom knowledge about the evaluation process to this setting. NO SCORE SELECTED Question Comments : Articulates how OT theories of occupation are applied within this practice setting NO SCORE SELECTED Question Comments : Articulates ways in which current evidence could impact practice within this setting. NO SCORE SELECTED Question Comments : PROFESSIONALISM Participates in experiences with enthusiasm NO SCORE SELECTED Question Comments : Takes initiative to maximize learning and uses time effectively. NO SCORE SELECTED Question Comments : Manages emotions professionally. Question Comments : NO SCORE SELECTED FIELDWORK RELATIONSHIPS Respects client rights and confidentiality. 62 NO SCORE SELECTED Question Comments : Uses effective communication with supervisor and other staff NO SCORE SELECTED Question Comments : Responds appropriately to supervisory feedback. NO SCORE SELECTED Question Comments : Takes initiative to ask questions. NO SCORE SELECTED Question Comments : Utilizes professional ethics NO SCORE SELECTED Question Comments : Wears suitable attire and is neatly groomed. NO SCORE SELECTED Question Comments : Is prompt and prepares for all fieldwork sessions NO SCORE SELECTED Question Comments : Turns in all fieldwork assignments promptly. NO SCORE SELECTED Question Comments : Adheres to facility rules and schedules. NO SCORE SELECTED FIELDWORK RELATIONSHIPS 63 Question Comments : COMMENTS (When completing this section, you may include diagnoses, age ranges, optional experiences, remediated problems, strengths and weaknesses.) Comments : Evaluation Score Summary Title: Primary Evaluation Score Weight Adj. Score 0.00 100.00% Required  ...

... Mind the Gap: Knowledge of Diastasis Recti Abdominis in Postpartum Women Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Jennifer Wiley, DPT Copyright October 5, 2020 By: Jennifer Wiley, DPT All rights reserved Approved by: Laura Santurri, PhD, MPH, CPH Committee Chair ______________________________ Lisa Borrero, PhD Committee Member ______________________________ Ruth Maher, DPT, PhD Committee Member ______________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program Chair, Interprofessional Health & Aging Studies University of Indianapolis ______________________________ Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis ______________________________ Running head: MIND THE GAP 1 Mind the Gap: Knowledge of Diastasis Recti Abdominis in Postpartum Women Jennifer Wiley University of Indianapolis MIND THE GAP 2 Abstract Diastasis recti abdominis (DRA) is a condition that involves a lasting separation of the abdominal muscles, found commonly in women after pregnancy. This condition leads to limitations in the physical and emotional well-being of women. The existing focus of research on DRA has been on diagnostic tools and treatment options. There is no evidence of investigation into the lived experience of women with this condition nor about the forms or topics of education provided by healthcare providers. The purpose of this study was to explore the lived experience of women who have been either self-identified or medically diagnosed with DRA. A deeper insight into the experience of women with DRA may serve to enhance the patient-provider interaction during this critical time. A qualitative methodology was utilized with a basic interpretive approach and convenience sampling. Each participant contributed her experience through a semi-structured, in-depth interview. Eight predominant themes emerged from the analysis of data: lack of patient knowledge, lack of education offered by providers, impact on self-image, desired education, sources of self-education, benefits of education, problems with available education, and patient recommendations for care. These themes help to create a picture of the experiences of each participant. The data collected identifies gaps in the patient-provider relationship and offers opportunities for improvements in available evidence-based knowledge. MIND THE GAP 3 Table of Contents Title Page ........................................................................................................................................ 1 Abstract ........................................................................................................................................... 2 Chapter 1: Introduction ................................................................................................................... 5 Background ......................................................................................................................... 5 Problem Statement .............................................................................................................. 6 Purpose Statement ............................................................................................................... 6 Research Question .............................................................................................................. 6 Significance of the Study .................................................................................................... 7 Chapter 2: Literature Review .......................................................................................................... 8 Prevalence ........................................................................................................................... 8 Diagnosis............................................................................................................................. 8 Impairments ........................................................................................................................ 9 Interventions ..................................................................................................................... 10 Areas for Improvement ..................................................................................................... 10 Chapter 3: Method ........................................................................................................................ 14 Study Design ..................................................................................................................... 14 Participants ........................................................................................................................ 14 Procedures ......................................................................................................................... 15 Data Management & Analysis .......................................................................................... 17 Chapter 4: Results ......................................................................................................................... 20 Chapter 5: Discussion ................................................................................................................... 32 References ..................................................................................................................................... 39 MIND THE GAP 4 Table 1 Participant Characteristics ............................................................................................... 48 Apendices ...................................................................................................................................... 49 Appendix A Recruitment Flyer ......................................................................................... 49 Appendix B Information Sheet ......................................................................................... 51 Appendix C Demographic Sheet ...................................................................................... 53 Appendix D Semi-structured Interview Guide ................................................................. 54 MIND THE GAP 5 Mind the GAP: Knowledge of Diastasis Recti Abdominis in Postpartum Women Diastasis rectus abdominis (DRA) is a phenomenon commonly found in females due to the separation of the rectus abdominis musculature during pregnancy (Stedman, 2006) and can be found in up to 68% of women immediately postpartum (Keshwani, Mathur, & McLean, 2018). Most cases will show a decrease in the distance between rectus abdominus muscles, or inter-recti distance (IRD), within 12 months postpartum (Sperstad, Tennfjord, Hilde, EllstrmEngh, & B, 2016). However, as many as 30% of instances may not resolve over time (Keeler et al., 2012). Factors that increase the risk of DRA include multiparity, cesarean sections (Spitznagle, Leong, & Van Dillen, 2007; Turan, Colluoglu, Turkyilmaz, & Korucuoglu, 2011), and history of abdominal surgeries (Spitznagle et al., 2007). Current evidence surrounding DRA includes prevalence, associated symptoms diagnostic techniques, and treatment and management options. Physical impairments associated with DRA include urinary and fecal incontinence (Benjamin, Frawley, Shields, van de Water, & Taylor, 2019), pelvic organ prolapse (B, Hilde, Tennfjord, Sperstad, & Engh, 2017; Lawson & Sacks, 2018), and impaired abdominal muscle strength and low back pain (Benjamin et al., 2019; Lawson & Sacks, 2018). Women are also at a higher risk for poor body image and lower quality of life (Benjamin et al., 2019; Nayir et al., 2016). Conservative and surgical management options have been studied and reported to be effective in reducing pain (Belogolovsky, Katzman, Chirstopherson, Rivera, & Allen, 2015), reducing IRD, and improving function in postpartum women (Benjamin, van de Water, & Peiris, 2014; Kamel & Yousif, 2017; Litos, 2014). Despite the association of significant physical symptoms and risk of lower quality of life with DRA, there are no studies that have explored the lived experiences with the condition. Investigating the lived experience associated with DRA can facilitate a deeper understanding of MIND THE GAP 6 the effectiveness of health education, direct better ethical care (Dierckx de Casterl et al., 2011), and improve compliance and utilization of treatment options (Jin, Sklar, Oh, & Li, 2008; Naidu, 2009). Problem Statement Despite a growing interest and demand for medical support for women's health issues (Dall, Chakrabarti, Storm, Elwell, & Rayburn, 2013; Daniels, Erickson, & Bornstein, 2018; Stenberg et al., 2014), there has been no investigation into the lived experience for women who have been diagnosed with DRA. A continuation of research on diagnostic tools and viable treatment options does not ensure that women with DRA understand the associated health risks and treatment options available. The lack of emphasis on women's experiences fails to examine if women are aware of current treatment options and if treatments are perceived as practical, sustainable, or even acceptable to those experiencing this condition. As such, there is a need to investigate the perceptions and experiences of postpartum women with DRA, specifically surrounding the education received about the diagnosis, associated risk, and treatment options. Purpose Statement The purpose of this qualitative study was to understand the experience of postpartum women who have been either self-identified or medically diagnosed with DRA from diagnosis through management or treatment of the condition. This study provides a more in-depth insight into the care and education for postpartum women with DRA. Research Question This study addressed the following primary research question and associated subquestions: MIND THE GAP 7 What is the experience of postpartum women with Diastasis Recti Abdominis regarding diagnosis, management, and/or treatment of their condition? o How do pregnant and postpartum women perceive the information and education they receive regarding the condition? o How do women seek information and education on the condition? Significance of the Study Gaining an understanding of the experiences of women with DRA, including their path and process of education, will help healthcare providers to deliver more effective, targeted educational programming and treatment. An improved educational delivery would allow women with DRA to make informed decisions, weigh risk, and have a voice in the selection of appropriate treatment. Improving the insight of healthcare providers into the experiences of women with this condition could facilitate the delivery of more effective education, which could result in improved patient compliance and outcomes. Definitions of Terms Postpartum. The period immediately following giving birth that can last up to six months (Romano, Cacciatore, Giordano, & La Rosa, 2010). Parity. A term to describe the number of times a woman has given birth to a fetus at a gestational age of 24 weeks or more (Tidy, 2019). Gravidity. A term to describe the number of times a woman has been pregnant (Tidy 2019). Multiparity. A term to describe a woman how has given birth more than once (Tidy, 2019). MIND THE GAP 8 Literature Review Prevalence DRA is a separation of the connective tissue between the rectus abdominus muscles, which occurs during pregnancy as the fetus grows. This connective tissue, also called the linea alba, softens as the abdominal wall distends, and the rectus abdominus muscles elongate and move laterally, causing a separation (da Mota, Pascoal, Carita, & B, 2014). DRA commonly affects 30 to 50% of women during the gestational period and in 60% of women six weeks postpartum (da Mota et al., 2014; Sperstad, Tennfjord, Hilde, Ellstrm-Engh, & B, 2016). While most cases will show a decrease in inter-recti distance (IRD) within 12 months postpartum (Sperstad et al., 2016), as many as 30% of instances may not resolve over time (da Mota et al., 2014). Ninety-six percent of all DRA cases occur at the level of the umbilicus, 68% occur above the umbilicus, and 40% occur below the umbilicus (Ponmathi, Jayalakshmi, & Sivakumar, 2016). Number of parity and history of abdominal surgery have been identified as factors leading to the incidence of DRA. In a study of 95 women between the ages of 19-24, Turan, Colluoglu, Turkyilmaz, & Korucuoglu (2011) identified the presence of DRA in 59% of multiparous patients compared to 2% in primiparous. A retrospective study by Spitznagle, Leong, & Van Dillen (2007) identified 281 of 541 patients as having DRA. Of those patients, 28 (9.96%) were nulliparous, and 253 (90%) were multiparous. 65.13% of patients with a history of abdominal surgery were also identified as having DRA. Diagnosis There are several ways to evaluate and screen for DRA, but agreement on criteria that constitutes a diagnosis of DRA has not been achieved (Barbosa, de S, & Coca Velarde, 2013). MIND THE GAP 9 Ultrasound, generally considered the gold standard for diagnosis of DRA, has limited use in the clinical setting due to costs and required training (Barbosa et al., 2013; Keshwani, Mathur, & McLean, 2018). However, caliper and finger-width methods, which are commonly used, have acceptable retest, inter-, and intra-rater reliability (van de Water & Benjamin, 2016). The identification of women with DRA is often missed despite knowledge of the high prevalence and ability to use reliable diagnostic tools (Freeman, 2013). When left undiagnosed, DRA can significantly affect a woman's quality of life, with most women reporting difficulty with daily activities (Benjamin, van de Water, & Peiris, 2014; Benjamin, Frawley, Shields, van de Water, & Taylor, 2019). Impairments DRA is associated with a variety of physical impairments. The severity of low back pain, as well as abdominal muscle weakness, has a relationship to the severity of DRA (Benjamin et al., 2019). A significant relationship was found in the presence of DRA in women seeking medical treatment for abdominal or pelvic region pain compared to those not seeking treatment (Parker, Millar, & Dugan, 2008). Pelvic organ prolapse and urinary and fecal incontinence are also associated with the presence of DRA in postpartum women (B, Hilde, Tennfjord, Sperstad, & Engh, 2017; Lawson & Sacks, 2018). Though these impairments can exist in the presence of DRA, studies have been unable to definitively identify a higher prevalence than in postpartum women without DRA (B et al., 2017; Chiarello, 2017). In addition to physical effects, DRA also leads to a higher risk for a lower quality of life (Benjamin et al., 2019) and poor body image in postpartum women (Keshwani et al., 2018). The increased IRD found in DRA contributes to lower body image and body satisfaction ratings (Keshwani et al., 2018). In a study using the Body Image Scale and the World Health MIND THE GAP 10 Organization (WHO) Quality of Life Scale Short Form, each sub-domain of quality of life was significantly affected by body image (Nayir et al., 2016). Despite these findings, a more in-depth look into the specific reasons for poor body image needs investigation. Identification of specific causes for poor body image and quality of life related to DRA could assist in a more comprehensive care plan for postpartum women. Interventions The associated symptoms with DRA have led to developments and improvements in treatment options. Surgical repair of DRA is the most common treatment offered to postpartum women diagnosed with the condition (Mommers et al., 2017). Unfortunately, surgical options are invasive and have a recurrence rate of DRA in up to 40% of cases (Hickey, Finch, & Khanna, 2011). More recently, advancements of conservative treatments offer women encouraging options. Conservative management of DRA includes physical therapy, exercise, and education on the maintenance of the condition (Opala-Berdzik & Dbrowski, 2009). Specific exercise treatments such as neuromuscular electrical stimulation and functional strengthening exercises have been found to reduce pain (Belogolovsky, Katzman, Chirstopherson, Rivera, & Allen, 2015) and reduce the inter-recti distance (Benjamin et al., 2014; Kamel & Yousif, 2017; Litos, 2014). The high prevalence, overwhelming symptoms, and a variety of diagnostic options for DRA should make identifying women with this condition and referring them to appropriate treatment a routine practice. Areas for Improvement Key components of maternal health include prevention, detection, and management of medical complications. Regrettably, interventions that promote health in postpartum women, MIND THE GAP 11 including referrals to appropriate management and treatment continues to be a missed opportunity among healthcare providers (Fahey & Shenassa, 2013). Reasons for the mismanagement of DRA may fall on both the treating medical providers as well as the patient. In a study by Gustavsson & Eriksson-Crommert (2020), the authors reported a lack of knowledge among physiotherapists and midwives on the relevance of increased IRD. The study also identified a need for clinical guidelines for the treatment of increased IRD along with better interprofessional collaboration on the management of the condition. In addition to the need for clinical guidelines, Gustavsson & Ericksson-Crommert (2020) reported an overall lack of evidence-based knowledge that is available to health care professionals. This lack of valid and reliable information pushes health care professionals to rely on the only accessible information, which has been from media and commercial based fitness coaches (Gustavsson & ErickssonCrommert, 2020). Since as early as 2010 there has been an increase in the use of internet-based resources among pregnant women (Lagan, Sinclair, & Kernohan, 2010; Lupton & Maslen, 2019). The information received is being used to help guide decisions during pregnancy, clear up confusion, and provide much-needed support (Lupton & Maslen, 2019). With a lack of evidence-based information on IRD and DRA available to health providers, it is not surprising that a greater percentage of Americans choose the internet as their first source of health information over any other source (Jacobs, Amuta & Jeon, 2017). The greatest concern with the use of internet-based sources for health information is the quality of information available. Though not specific to DRA, a study by Burton et al. (2020) on strategies for prevention and treatment of urinary tract symptoms in women on social media, it was found that only one-third of the evidence found on social media lacked scientific evidence MIND THE GAP 12 and the majority did not include level 1 evidence. In the same study, it was found that the least mentioned treatments for many of the diagnoses were the options with the greatest successes. In addition to a lack of evidence behind the information provided on the internet, there is also a concern about the negative impact of internet-based sources such as online support groups. In the qualitative study by Lupton & Maslen (2019), participants mentioned the caution needed when joining online support groups due to the tendency of judgment and bullying. In addition to patients using internet-based sources for their health education, many postpartum conditions experienced by women are seen as expected symptoms and they choose not to seek medical help due to embarrassment or wasting the doctors time (Geynisman-Tan, Taubel, & Asfaw, 2018; Rowlands, Loxton, Dobson, & Mishra, 2015). If appropriate education about these expected symptoms was provided, feelings of undue embarrassment could be mitigated. The early postpartum period has been termed the fourth trimester and it has been noted that this is a crucial time for much-needed education and medical help which can improve the health and wellbeing of women (Tully, Stuebe, Verbiest, 2017). Without appropriate care and intervention in that fourth trimester, many women are left with confusing symptoms, which may compound one another thereby necessitating timely multidisciplinary medical care (Tully et al., 2017). Improving the identification of DRA and referrals to appropriate treatment providers begins with education. Patient education leads to patient empowerment (Yeh, Wu, & Tung, 2018). Patient empowerment is a concept that allows the patient to take an active role in their health by becoming part of the healthcare decision-making team (Castro, Van Ragenmortel, Vanhaecht, Sermeus, & Van Hecke, 2016). Patient empowerment requires a patient-centered approach to care and communication through focused patient education provided by the MIND THE GAP 13 healthcare worker (Funnell, 2016). Structuring patient education to emphasize the impact of the condition (Funnell, 2016), the risks associated with that condition, and the treatments available improve patient empowerment and result in better healthcare outcomes (Jin, Sklar, Oh, & Li, 2008; Yeh et al., 2018). Improving the provider-client interaction through education can promote a health care model that encourages individuals to live their life to its fullest potential (Fahey & Shenassa, 2013). Developing an understanding of the experiences of patients cultivates insight into the knowledge they have gained through provider interaction and patient education (Dierckx de Casterl et al., 2011). Qualitative research, a method of understanding how people experience their life (Creswell & Poth, 2018), has strengthened our understanding of patient noncompliance specific to urinary incontinence and explained limitations in the generalizability of outcomes (Doshani, Pitchforth, Mayne, & Tincello, 2009). Unfortunately, qualitative research specific to DRA is not currently available, but the value of qualitative research on urinary incontinence in women can translate to other urogynaecological diagnoses such as DRA. Qualitative research helps participants and researchers construct an understanding of participant expectations and priorities related to quality of life. Obtaining a detailed understanding of how participants interpret quality of life and functional limitations using qualitative research could better define improvements in outcomes (Doshani et al., 2009). Improved patient compliance and satisfaction through effective patient education can be assessed through understanding the lived experience of the patient. The understanding of the experience through the patients' view may help direct effective care by healthcare providers (Dierckx de Casterl et al., 2011). MIND THE GAP 14 Method Study Design A basic interpretive qualitative methodology was used to gain an understanding of how women, diagnosed with DRA, interpret their experience with this condition. A basic interpretive design allows researchers to focus on the personal interpretation of patients living with a diagnosis, such as DRA (Merriam, 2002). The purpose of this focus was to collect data on the uniqueness of individual experiences related to the diagnosis of DRA (Creswell & Poth, 2018). Data collection took place from February 2020 to July 2020. Prior to participant recruitment, the study received approval from the PCOM Division of Research Internal Review Board (Protocol #H20-002x). Participants Convenience sampling was used to recruit adult female participants with a history of medically diagnosed DRA or self-reported symptoms consistent with DRA. For this study, a sample size was desired to reveal themes that illustrated depth and variation and reflected the experiences of the participants (Corbin & Strauss, 2008). As described by Morse (2000) sampling for qualitative studies does not have hard and fast rules. The sample size was determined with the consideration of a study's scope, subject matter, and quality of data. Due to the specifics of this population and the goal of the study, 13 participants were included. All participants were at least eight weeks but no more than three years postpartum to ensure time for the natural healing of DRA (Coldron, Stokes, Newham, & Cook, 2006). The postpartum cut-off allowed recruitment of enough participants for an effective study and reduce the distortion of memory recall of the events during and after pregnancy over time (Lacy & MIND THE GAP 15 Stark, 2013). Exclusion criteria included current pregnancy for the participant to reflect on experiences before, during, and after giving birth and non-English speaking women. Procedures Recruitment. Recruitment took place in a variety of settings such as outpatient physical therapy practices with pelvic health therapists, pediatric physical therapy practices, online support groups for mothers/expecting mothers, and local colleges and universities. The primary investigator visited the physical settings or posted to the online forum and asked the practice manager to post a recruitment flyer (see Appendix A). The initial postings of the recruitment flyer elicited 22 responses from interested women. The primary investigator contacted all who express interested to explain more about the study and answer any questions from the potential participants. Ultimately, 13 participants were able to participate in the study, as nine women were currently pregnant and not included. Interviews were scheduled with each participant and information sheets were provided to give a written description of the study (see Appendix B). Informed consent. To reduce the collection of identifiable information, the primary investigator received a waiver of written informed consent from the IRB. A participant list, including names, participant pseudonyms, and contact information, was kept in hard copy form in a locked file cabinet separate from all study data to facilitate member checking following transcription of the interviews. At the start of the interview, each participant was given another copy of the information sheet (see Appendix B). This document explained in detail the purpose of the study as well as the procedures that would occur (Marshall & Rossman, 2011). Specifically, the document included the identification of the researcher, sponsoring institutions, the purpose of the study, and the risks and benefits of participation. The participants were also advised that they had the right to MIND THE GAP 16 withdraw from the study at any time and/or request that parts of the transcript not be used (Creswell, 2014). Verbal consent from each individual was obtained before the commencement of the interview, following the opportunity for each participant to ask any questions about the study. Data collection process. Following verbal consent, each participant completed an intake form (see Appendix C) with the help of the primary investigator. The form did not include identifying information and was used to collect information about past medical history including a history of all pregnancies and births. An assigned pseudonym was used to match the intake form with the focused interview. The intake form was also used to gather demographic information to increase the trustworthiness and transferability of the study (Creswell & Poth, 2018; Henderson & Rheault, 2004). Following verbal consent and all questions being answered, semi-structured in-depth interviews were conducted either face-to-face or via a Zoom (Zoom Video Communications Inc., San Jose, CA) virtual meeting with each participant and the primary investigator. The purpose of the one-on-one, in-depth interview was to gain an understanding of the personal, lived experience of participants self-identified or medically diagnosed with DRA in their own words (Marshall & Rossman, 2011). Face-to-face interviews were preferred due to the likelihood of a more controlled environment that allowed for the collection of both verbal and nonverbal communication (Creswell & Poth, 2018). Interviews via Zoom virtual meetings were used when participants could not engage in an in-person interview due to living in a different state and the social distancing mandate due to COVID-19. Outside distractions were minimized by completing interviews in a quiet, private location to preserve participant privacy. MIND THE GAP 17 Each interview was audio-recorded to ensure the accuracy of transcription and coding of data. To direct the interview, the primary investigator utilized a semi-structured interview guide (see Appendix D) containing general, open-ended questions designed to facilitate discussion around the participants' experiences with DRA. Probing and follow-up questions in response to information shared by the participant were also used to gather comprehensive information. The length of interviews ranged from 15 to 45 minutes when the participant did not reveal any new information. Following the interview, the participants were given the contact information for the primary investigator, should they have any additional questions or concerns. Finally, the primary investigator informed each participant that a follow-up via email would be sent at a later date with the themes generated from their interview. The purpose of the email was to have participants review the themes as a form of member checking to ensure that an accurate representation of their experience had been captured. Steps to ensure confidentiality were reiterated to the participant in addition to how the information will be disposed of after 5 years following completion of the study. Throughout the interview, the researcher kept field notes, as this helped to identify researcher bias, document researcher interpretation, and note data on non-verbal communication (Creswell & Poth, 2018). The use of this type of documentation helped to ensure the trustworthiness of the study as well as increase rigor by reducing bias within the data (Henderson & Rheault, 2004). Data Management & Analysis Upon completion of each interview, the primary investigator along with three other investigators transcribed each audio recording using audio to text service followed by an extensive review to ensure accuracy of the transcription. Transcription of the interviews facilitated the investigators' immersion into the data (Halcomb & Davidson, 2006), which allows MIND THE GAP 18 the investigator to thoroughly explore and identify specific codes and themes that represent the experiences of the participant (Creswell, 2014). All transcripts and audio recordings were deidentified and assigned corresponding pseudonyms and stored on the participant list, which was stored separate from all other study documentation. The de-identification of data helps to ensure the privacy of the participants (Creswell, 2014). The primary investigator then reviewed each transcript multiple times while reviewing field notes documented during and after the interviews to become more familiar with the data. This familiarization allows the investigator to become immersed in the data to prepare for coding (Marshall & Rossman, 2011). The primary investigator utilized open coding to identify words and phrases that were notable (Creswell, 2014). The notable words and phrases were assigned a "code," representing essential information related to the research questions (Creswell, 2014). A crosscheck of the codes was completed by enlisting two additional researchers, one who is a board-certified specialist in women's health physical therapy, and another with an extensive background in qualitative research. These additional researchers read transcripts 0652 and 0569 respectively and coded the data independently to help build the foundation for a codebook. Crosschecking of the codes allowed for code agreement, increased trustworthiness of data, and development of a codebook (Creswell, 2014). Once both the primary investigator and additional researchers completed coding, a consensus was achieved about the codes. Axial coding was performed by the primary investigator to eliminate redundancies and group the codes into broader categories (Saldana, 2016). The primary investigator developed a codebook to organize the codes and offer code definitions and sample quotations from the transcripts. From the codebook, commonalities among the code definitions and categories were MIND THE GAP 19 identified and used to develop overarching themes that best reflected the data (Saldana, 2016). All coding and codebook development was stored and managed using a password-protected Microsoft Excel spreadsheet. Following theme generation, member checking was performed. Two participants sought clarification on the theme generation process. Once themes were reviewed and quotation examples were sent to them via email all were in agreeance. Member checking ensured an accurate representation of the participants' experience and increased the trustworthiness of data (Marshall & Rossman, 2011). Throughout the data analysis process, an audit trail was also be maintained in the Microsoft Excel spreadsheet. The audit trail included details of the decision-making process as well as all steps performed. This process increased the trustworthiness of the study by clearly outlining all steps completed within the study and documenting the researcher's thoughts and potential biases throughout (Creswell & Poth, 2018). Due to the researcher's background with the topic of DRA, a reflexive journal was maintained to reduce additional bias. Reflexive journaling can assist in accounting for how a researcher's background may have affected the study (Creswell, 2014). Rigor/Trustworthiness The utilization of current research processes and logic strengthen the arguments made by research findings and increase the rigor and trustworthiness of qualitative studies (Ryan-Nicholls & Will, 2009). Rigor and trustworthiness were addressed in this study by strategically influencing the credibility, transferability, dependability, and confirmability (Henderson & Rheault, 2004). MIND THE GAP 20 Consistent with recommendations by Henderson and Rheault (2004), reflexive journaling and use of an audit trail enhance credibility, dependability, and confirmability in the study. The use of reflexive journaling and audit trails allows the researcher to document personal feelings, perceptions, biases, and steps taken throughout the study and data collection (Creswell & Poth, 2018; Henderson & Rheault, 2004). Credibility is also influenced by member checking to ensure the accuracy of transcripts and the use of a second researcher to triangulate the data analysis process (Creswell, 2014; Morse, Barrett, Mayan, Olson, & Spiers, 2002). Finally, transferability is strengthened by collecting a detailed history of each participant, which allows for future comparisons to a larger population of postpartum women experiencing DRA (Henderson & Rheault, 2004). Results Thirteen participants were interviewed for this study. Demographic information is described in detail in Table 1. Participant names were removed and replaced with pseudonyms to protect confidentiality. Eight participants had a medical diagnosis of DRA, and five were selfdiagnosed by matching the symptoms associated with DRA as outlined on the recruitment flyer and study information sheet. Recruited participants had a mean age of 31.3 (3.65) with an average gravida 2.3 (0.5) and para 2 (0.3). Of the 26 total births among participants, 16 were of natural delivery and 10 were cesarean, with only one multiparous birth. Three interviews were conducted face-to-face, and due to social distancing mandates, ten interviews were conducted using the Zoom video conferencing system. The duration of interviews ranged from 15-45 minutes. Eight major themes emerged from the interviews: lack of patient knowledge, lack of education offered by providers, impact on self-image, desired education, sources of self- MIND THE GAP 21 education, benefits of education, problems with available education, and patient recommendations for care. These themes help to create a picture of the experiences each participant had regarding living with DRA. Theme 1: Lack of Patient Knowledge The participants expressed their frustration throughout the interviews about not understanding what they were experiencing. These frustrations became a commonality across all participants. The inability of many participants to explain their understanding of DRA made it a predominant theme. For the participants who were medically diagnosed with DRA, it meant they were participating in life with a diagnosis but without an understanding of what that diagnosis meant. Across all participants, there was a severe lack of knowledge behind associated symptoms of DRA, understanding of the cause of DRA, the prognosis of the diagnosis, and viable treatment options. Brit commented regarding the symptoms she was experiencing, Um, I just kept wondering like, what is going on? This doesnt seem right. Rebecca stated: Whats my, you know [body], doing? What is my, you know, just this constant fear griped in the back of your head and you are like, okay, am I feeling this because I am feeling it or am I feeling this because Im making it up. Of the five participants that were self-identified as having DRA, the lack of knowledge of DRA limited their ability to recognize symptoms and accept the need for help. Because I didnt know anything about it was the reason Jamie gave for accepting the symptoms she was experiencing as normal. Tara commented, Well, as far as my health goes, I just kind of, I kind of rolled with it. I feel like it is what it is. MIND THE GAP 22 Five of the participants work in the health care field, either as a nurse or physical therapist and all felt that though they were more comfortable with their knowledge of the body they never expected or suspected DRA. Jennifer commented, I think, I guess I just know being a PT, I could probably figure it out but I didnt know. Rheema stated: I thought I would know that thats one thing to bring up at your doctors appointment after you have the baby. If I didnt have the background I do, and the coworkers I had, I would have never known to do something about it. I wouldnt have even thought to bring it to a doctor. I just passed it off as part of having a baby. When asked about knowledge of prognosis and available treatments for DRA, none of the participants were able to verbalize their understanding of a prognosis for DRA. Five simply reported, I have none. Most reported that DRA was not a topic of discussion with healthcare practitioners, friends, or most other sources. As for treatment options, two were able to identify surgery, but details about the surgical option were not understood. Jamie: I feel like if I had known immediately thats what it was. I could have gone, you know, to physical therapy and, you know, gotten the problem fixed a lot earlier, um, than waiting so long to even know what it is and more about it. From the discussion on existing knowledge of DRA, conversations lead the emergence of another theme that expanded on what information is provided by healthcare practitioners in prenatal and postnatal appointments. Theme 2: Lack of Education Offered by Providers Participants reported that the education topics and focus of conversation during pre- and postnatal appointments included birth control following birth, preeclampsia, breast-feeding, and MIND THE GAP 23 postpartum depression. When specifically asked about the education received about DRA, all thirteen participants stated there was no education before the patient questioning their symptoms. Along with education topics, provider response to patient concerns was discussed. Many of the participants felt that the providers response to their expressed concerns was dismissive. I think he wrote me off because he knows Im a PT Sarah stated. Similarly, Tara stated about her physician, Um, no [investigation of symptoms]. But this one, I think they chalked it up to me already having two [kids] and they said, Oh, well, youre a pro at this, so you know, whatever you need, whatever questions you need, um, you know, you kind of already understand. Sharon stated, My doctor told me it [my protruding belly] was just excess, you know, body fat. Annalise said, Um, which again, looking back and like, man, I cant believe my doctor didnt mention this while I was pregnant. And I literally said, it feels like my muscles are ripping apart. In response to voiced concerns, many participants stated their providers qualified symptoms as part of having a baby. Myrna stated, they dont, they dont discuss the actual stuff that happens with it [DRA]. Brit: So, I would probably, I would really probably argue with them and say, you know, you all really should start, you know, learning about this and just be more aware of it and instead of just dismissing it and saying that its just a part of motherhood or having kids because it shouldnt be like I should have to give up. Like I said, I all but got laughed at when I would bring it up in the doctors office and physical therapist, She was just like, are you seriously wasting my time with this? I mean, she didnt say that, but that, that was the vibe that I got. Theme 3: Impact on Self Image MIND THE GAP 24 The lack of existing knowledge and provided education as described by the participants impacted the lives of each woman in different ways. The symptoms, whether identified as associated with DRA or not, also left an impact on body image, self-confidence, and self-doubt. Connie: The way your stomach looks. Um, I look pregnant still, even when I diet and lose weight, its still protruding and sometimes even worse to the point where I look better heavier. Sharon stated, Youve got like your stomach sticking out because you have excess weight or this problem, and thats where I was trying to figure out is this really a thing or do I really just have excess weight? In regards to the participants body image with DRA related symptoms, Jamie stated, I dont really care for it [her body]. While Myrna said: I still look pregnant. After pregnancy, Ive struggled extremely. It makes you conscious about yourself, and I dont even, like makes you feel ugly, not like yourself. And like looking at myself in the mirror, especially when you have two girls, puts more self-doubt on the body and self-image. I dont want them [daughters] to have a negative body image. The lack of knowledge and understanding of DRA also caused confusion as to what the participants should physically do. Jamie: Like it was almost scary I guess, cause I was like, am I going to make, I dont know if I can make this worse, is this going to make it better? So Ive kind of erred more on the side of caution. Theme 4: Desired Education MIND THE GAP 25 The impact that the lack of knowledge and understanding of DRA had on each participant was very clear throughout the interviews. Within the accounts, each woman identified the type of information and education they desired. Reports of desired education included the acknowledgment that DRA could occur, the prognosis of the diagnosis, treatment options, and referrals to other providers who could help with the management of the diagnosis. Jamie stated, I think healthcare providers should inform women on what it is and you know, how to look for it. Cause, um, I mean, were not told anything. I think I have a lifelong fear about that [DRA] happening again (Linda). Rheema: Yeah, I think that, I think healthcare providers should inform women on what it is, how it happens and you know, how to look for it. Signs of it, like to know what it is. Also, what can be done for it. And if they [physicians] dont know, then they need to refer to a really good pelvic health therapist or orthopedic therapist that treats postpartum issues. Rebecca: Um, just giving the basics. This is what it is. There is treatment. If you notice, you know, these kinds of things, go see someone, kind of like we do for back painI dont subscribe to like a blanket prescription for everyone, but if you have this, you know, see someone, maybe you shouldnt be doing crunches at 25 weeks pregnant. Brit: I say that they should have maybe like brochures or something. I mean, they have brochures for everything at the doctors offices. Do something about this and make a brochure, of some sort to give women some type of knowledge of what is going on. Along with education on the possibility of DRA occurring during pregnancy some women wanted specific information. I would have like to have heard that [exercise options] versus the only way to fix it, a surgery (Connie). She also added, I guess knowing what MIND THE GAP 26 exercises not to do that could aggravate it because I was doing those and it did make it worse. Another suggestion for providers was, Whatever the diagnosis is, seeing people postpartum and then getting them to the right person (Rebecca). Finally, Rheema suggested, So, I guess the other thing would be to get a support group. Connie highlighted the problems that can arise when proper education is not given and women are left searching for knowledge: I think the exercise, knowing the exercises, um knowing I guess the prognosis. Um, what if it can be treated and what you can do because there is a lot of information out there and not all of them are on the same page, so to speak. Um, some say there is nothing you can do. Some others say you can correct it with certain exercises and diet and programs. Um, so I guess just letting them [women] know some good information as to what they can do or if they can do anything about it. Theme 5: Sources of Self-Education Within eight of the interviews, the women described that their need for education on DRA led them to investigate sources outside of the primary medical provider caring for them during and soon after birth. All cited use of the internet led by Google searches. I was Googling what kind of doctor do you see for diastasis Sharon stated. Myrna said, I started doing more research recently to try to figure out what kind of doctor I should even see for this. In addition to Google searches, some of the participants looked into blogs, Pinterest, and specific videos for treatment ideas. I found blogs and different websites and like Pinteresting exercises for diastasis recti (Sharon). Jennifer spoke about other women who have experienced DRA becoming sources of information for them. MIND THE GAP 27 But one [coworker], um, does have a friend that has it, a lot of kids, um and then she became pregnant, so. And she was more fresh out of [Physical Therapy] school, so she graduated three years ago. Im like, hey, whats, what are you learning nowadays in school? Um so seeking out her and then I have an old classmate that, um, I knew she had one [DRA] and so just kind of asked her about hers and what shes doing and basically she just still has one [DRA]. Rebecca, a participant with a background in physical therapy, turned to research journals and Facebook support groups. I dove into the Journal of Physical Therapy, and other scholarly journals, mainly physical therapy based. I spoke with a mentor who does a lot with womens health. I used ol Google. I joined Facebook support groups but quickly realized all the bad information on those forums. Theme 6: Problems with Available Education For participants, arming themselves with information on DRA was important. This selfeducation came with reports of contradictory information and problems with access to resources. Jennifer stated: I just kind of started looking at YouTube videos and so I kinda, I would look at the credibility of the person doing it, and then I just started looking at all kinds of different videos and saw what they had in common. Cause some things like, you know, they would say do a crunch. Im like, okay, well she just lost her credibility, but you know, she had some other activities she was doing and Im like, okay, that one is good. MIND THE GAP 28 Rheema stated her concerns about online support groups. I joined Facebook support groups but quickly realized all the bad information on those forums. Way too many people who are not specialized in the area giving bad advice. Similarly, Rebecca reported: I would probably honestly caution people against the online support groups because theyre, if you dont have a good confidence level and understanding, theres a lot of negativity, there is some good things, but theres a lot of, dont do this, you know, Oh youre doing that? Dont. Thats terrible, you are going to hurt yourself. In addition to concerns with the credibility of information and contradictory advice, some of the women expressed the challenges they had with accessing proper care. but trying to find someone that understands it and can treat it in, in a way thats not just to do some contractions and it will get better (Rebecca). Sharon: I wanted to talk to somebody about it to understand exactly what did I need to do and for how long. When I called, they no longer were doing that. So, from that point Im just like, well, I cant, I dont, its probably not going to be covered by insurance. So thats what going to probably be $100 a visit. And then where do I need to travel? And then with my time for kids and family and Ive worked full time, it was just kinda like, okay, well never mind, well have to figure something out on my own or do more research and just make, make myself do the stuff at home. Lisa: With my insurance, I think that actually made it a challenge. So I had to find a physical therapist in my network and, um, right. I had to get a referral to basically one of the major hospitals in the area. Um, but finding, and appointment was like a month down the line. Brit recalled a friend who had surgery to repair her DRA: MIND THE GAP 29 I have a friend who she, hers was really so bad that she ended up having the, having to have this surgery and her, I mean, she had to pay completely out of pocket. And you know, I mean, it, it did affect her. I mean it affected everything or like, I mean, she was having issues because of it and a lot more debilitating than my issues. Um, and so, but no one, they know that like her insurance wouldnt cover the surgery. So it was all out of pocket. While many of the participants cited access to quality care and education, Brit was particularly discouraged at the need to fight for her care. I mean, I know this sounds mean, but its like if I want to have a fit body or if I want to feel good like I should be, I shouldnt have to fight for something, you know like there is something wrong, theres something different. My stomach is different. Theme 7: Benefits of Becoming Informed Despite the reports of barriers to improving their knowledge of DRA, four of the women were able to describe the benefits of the education they did receive. Rebecca: I think honestly, having the diagnosis and how, like how many fingers and all that was helpful to knowSo again, youre not making people fearful cause I think thats, theres enough fear around pregnancy. We dont need to add to it. Benefits of knowledge were reported by those that were able to find education either early in their postpartum journey or before their next pregnancy. Jennifer shared about having some DRA knowledge, like I think its just like that talking point in that connection that you can talk to other people about it. Myrna stated that understanding DRA allowed her to, better assess the situation and start early prevention. Sharon similarly stated, I feel like Im more aware, a conscious thing for me MIND THE GAP 30 to always be thinking about. So, I had a very, I had a better understanding about it and what I needed to do. Additionally, Rebecca reported that her understanding of DRA helped her outlook on the diagnosis. Uh, um, I would say that it is not the end of your life. Right. So many people think Ill never do X again. Or, you know, Im never going to have that. Im always going to have that like pregnant belly look. But, my ultimate, you know, is finding someone who is addressing your concerns. If theyre not addressing your concerns, I dont care what their title is, find someone else. And so then that being said, you have to tease out whats your goal. Theme 8: Patient Recommendations for Care Each participant described their experience with healthcare providers as well as how the education they received has played a role in their outlook on the diagnosis of DRA. In addition to their own experiences, each woman was very particular in what they would recommend for the future of DRA care. Sharon: I think it definitely needs to be that it needs to be a part of the education when you're pregnant and after. And I've talked to women about it, but it's even, it's funny because talking with to women about it, most of them don't know what I'm talking about it'd be nice if more people knew about it and it was something that was actively, um, treated because I'm sure more people actually do have it than they realize. I'd be curious if there would be anything that you can do during pregnancy that could help prevent, or is this just something that is going to happen no matter what? If there is something that you could do during, I would like to tell them that, but if this is something that is just going to happen no matter what, how to seek treatment and what to do immediately so that they're MIND THE GAP 31 not three years down the line like I am and going, okay, this really is a problem. Just know that this is a possibility and what to do when you're done. Even just to go get checked. Don't just assume you're fine like that. Two participants touched on the need for women to become their own advocates and continue to ask for help until it is received. Rheema: I would say that you should never be ok with well you had a baby, you should feel this way. That is never ok. Pain is never normal. Even if it is an expected effect of pregnancy it doesnt mean you shouldnt have care for it. There are so many wonderful resources out there that are extremely knowledgeable and helpful and there needs to be more referrals to them. Also, unfortunately, never rely on a health care practitioner to give you all the information. So do not hesitate to ask about every little thing. And dont let anyone make you feel bad for asking for help. If one person will not help you, ask another until you get the help you deserve. Myrna: Be a normal, um, a normal procedure for your post follow-ups or something like that. Talk about it and ask questions (with provider). And then after pregnancy, after you know, the typical six weeks, seek out a PT person. Or, when they go in for the postpartum check ask their provider if they can check for it. Yeah, you got to advocate for yourself. If you feel like somethings up, like you really have to sometimes push the doctors to listen to you. Because they just brush it off. I will definitely tell other people advocate for your own selves and if they feel like somethings wrong speak about it. Don't feel negative. Another participant stated the need for education to be a regular part of postnatal care: So, if I had known about it before, I would now tell them, make sure you follow up with your provider and ask them specific questions in regards to that. I think that it should be, MIND THE GAP 32 at the very least, they give it as much attention as birth control after having a baby before you leave the hospital. I think a, just getting the word out that it's something that happens, but putting it maybe in a list of things (Tara). There was a significant amount of consistency among all of the participants interviews. They each described similarities in their interactions with healthcare providers as well as concerns with the available information and education provided. All participants spent a large amount of time during the interview outlining the particular recommendations they have to improve future experiences of women living with DRA. The themes identified through the coding process paint a very specific picture of the experiences these women had. Discussion During pregnancy, a womans body will undergo many changes that they may not understand. Women lean on respected sources to deliver information and education about pregnancy, labor, birth, and postpartum periods (Lagan, Sinclair, & Kernohan, 2010; Verbiest, Tully, Simpson, & Stuebe, 2018). Many times, the information a woman receives can help to direct decisions made about her and her childs care (Lagan, Sinclair, & Kernohan, 2010). If a woman experiences something during this time that is unexpected or unfamiliar, the outcome can affect each woman differently with possible long-term effects on her life (Verbiest, Tully, Simpson, & Stuebe, 2018). The investigation into the lived experience of a woman with DRA exposed themes of lack of patient knowledge, lack of education offered by providers, impact on self-image, desired education, sources of self-education, benefits of education, problems with available education, and patient recommendations for care. These findings may help healthcare providers develop better education and intervention strategies to assist women with this condition. MIND THE GAP 33 The perceived lack of existing patient knowledge about DRA limited the participants ability to identify symptoms, understand changes to their body, and limited their ability to acknowledge the need to seek medical advice. Similarly, a study investigating womens knowledge of basic pelvic floor disorders revealed that less than 50 percent of pregnant women had proficient knowledge of such disorders (Geynisman-Tan et al., 2018). A study by Verbiest, Tully, Simpson, & Stuebe (2018) also found that many women feel the presence of large gaps in clear directives on postpartum life. It was reported that women in the study felt providers sparked questions but further information was not provided (Verbiest et al., 2018). The lack of education offered by providers identified in this study aligns with a recent study by Gustavsson & Eriksson-Crommert (2020) that reports a lack of available evidencebased information, a lack of interprofessional collaboration for the management of increased IRD, and a lack of existing knowledge base on the condition by health care professionals. The possible insufficient knowledge base among the health providers left the participants feeling dismissed and as though their symptoms were a normal outcome of pregnancy that should be accepted. Similar findings were reported by Geynisman-Tan et al. (2018) among postpartum women experiencing incontinence. Women in the study by Verbiest et al. (2018) also reported that rather than receiving information and advice they received judgment. The results of this knowledge deficit resulted in reports of poor body image, low selfconfidence, and confusion. In a similar study, reports of a lack of knowledge caused women to be reluctant to seek care and resulted in women experiencing very isolating and distressing postpartum experiences (Geynisman-Tan et al., 2018). Body image and emotional well-being are areas affected by the postpartum time that many women feel could be improved with timely MIND THE GAP 34 education (Verbiest, Tully, Simposon, Stuebe, 2018). Unfortunately, these are topics of conversation that either never happens or happens too late (Verbiest et al., 2018). Uncertainty, confusion, and concern from the lack of knowledge and education left all participants in this study searching for information. That search was for specific answers on symptoms, prognosis, and treatment options for the condition. Every participant reported their search for information led them to internet-based sources due to the limited or lack of information provided by medical professionals. Sources that were cited included Google, Facebook, Pinterest, research articles, and online support groups. Though every participant reported, utilizing online sources many expressed their concerns with the quality of information available as well as the negative impact online support groups had. Consistent with the participants in this study, less than one-fifth of the women in a study by Lagan et al. (2010) reported trusting eighty percent of the information found on the online. The respondents also reported a concern that the information was wrong or misleading but despite the distrust, 100 percent of women surveyed continued to search the internet for health information. In another study, by Rowlands et al. (2015), a population of Australian women reported that womens health information found on social media came with hesitation toward the credibility and wiliness to accept the value of information. This perceived hesitation did not stop the frequency of women access internet-based information. The concern with online support groups identified in this study is a shared finding from a past qualitative study that investigated how women use digital technologies for health (Lupton & Maslen, 2019). Lupton & Maslen (2019) illustrated a limiting contribution of these groups as being a potential for bullying and negative judgment. MIND THE GAP 35 Despite the concerns with the accuracy and trustworthiness of the information from internet-based sources, women in this study perceived that receiving the information benefited them by either reducing anxiety, helping to clear confusion, confirmed suspicions, and even empowered them to seek additional help from health providers. Similarly in a qualitative study by Lupton & Maslen (2019), the participants reported information gained from online sources filled gaps in information or explanations of illnesses and that the readily available information gave them a sense of empowerment to decide if their concerns were valid or not. The reports of empowerment by some of the participants in the study were expressed as an outcome of gaining knowledge about their situation despite the source it was retrieved from. Though some participants did not report gaining this same sense of empowerment, all participants did report recommendations for women to become empowered to receive the needed education and care for DRA. The recommendations included encouraging women to push for answers about the symptoms they are experiencing, seek many health providers until appropriate care is found, and investigate available resources including the internet to encourage open communication between patients and health providers. Communication between patients and health providers is a key concept of patient-centered care (Verbiest et al., 2018). Doctor-patient communication specifically has been identified as having a direct effect on patient empowerment (Oh & Lee, 2012). The same study by Oh & Lee (2012) reported that a patients sense of empowerment strongly and positively relates to their intention to communicate with a doctor. Patient-centered care has been proven to increase patient empowerment by including the patient in decision making and results in better health outcomes (Castro et al., 2016; Funnell, 2016; Jin et al., 2008; Yeh et al., 2018). Limitations MIND THE GAP 36 Limitations in research play a role in any study conducted. In this study, researcher bias played a role due to the primary researcher experiencing DRA as well as being a physical therapist. Strategies such as reflexive journaling, audit trail, and triangulation were taken to reduce bias during data collection and analysis. Using convenience sampling the goal was to recruit a wide variety of participants that could contribute information from very different experiences. Ultimately, the sample did not produce a large variety of geographic locations within the United States or from other countries, which may offer different insights into other approaches to the education and care of women with DRA. Implications for Practice and Future Research Despite the stated limitations, this studys results drew much-needed attention to the education and knowledge women receive on the topic of DRA. The insight gained can be useful to health care providers and women who are or may become pregnant. Health care providers realize the need for a wider breadth of pre- and postnatal education and take steps to better educate themselves as well as their patients on the conditions and possible support available. Women may be empowered to become their own advocate when it comes to approaching providers about symptoms they are experiencing and questions they have previously been too scared to ask. The insight on the positive outcomes that education had on the mental health, body image, and overall comfort level with the condition of DRA may push health care providers to improve their communication with patients. Providers may also see the importance of including education about DRA as a standard component of the information provided to pregnant and postpartum women. Future research into the experience women have surrounding the diagnosis of DRA is needed to expand on the themes identified in this study. A mixed-method study could investigate MIND THE GAP 37 the experiences of women in a broader geographic area and compare the education received to the demographics of patients. This could give insight into health disparities among different populations. This type of study could also identify best practices that may be in place not only in the United States but also in other countries. A quantitative study could help to identify the effects modes of education in the prenatal period has on a womans knowledge of DRA and their pursuance of treatment if diagnosed. Perhaps a simple brochure about DRA provided to a patient during a prenatal appointment gives just enough information for a woman to identify her DRA symptoms and seek medical help. This would open communication between the patient and her provider to discuss possible options and result in an appropriate referral. An additional qualitative investigation into how the mode of education influenced the patients self-perception after DRA diagnosis could give providers an even deeper insight into providing better care. Additionally, another study to identify how access to services such as pelvic health physical therapy or other types of physical therapy may limit the amount of education on DRA provided during pre- and postnatal appointments. Perhaps obstetric-gynecologists are not educated on the services other providers offer for this condition or the lack of providers in the area impact the amount of education that is provided about treatment options. The concept of patient empowerment was evident across all participants in this study. The most common recommendation from each of the women was to express to other women how important it is to become their own advocate when it comes to their health. Each participant expressed that women should not wait for the information to come to them and they should push for the knowledge and understanding they need about all symptoms experienced. A future study MIND THE GAP 38 that investigates the perceptions patients have on empowerment and self-advocacy as it relates to health could help evolve the patient-caregiver relationship. Conclusion A woman navigating her way through pregnancy and birth experiences many changes within her body. Those changes could be short-lived or could live with a woman for the rest of her life. Education not only serves as a conduit of knowledge, but it also serves to provide comfort and understanding of a new normal. This study provides the insight that for women experiencing DRA the amount and type of education they receive is important. It can develop self-advocacy, affect self-perception of health status, and provide direction for possible treatment options that were otherwise unknown. It is not only the responsibility of health care providers to deliver that education but it is also the responsibility of patients to take an active role in their care and seek help. This study identifies potential gaps in the patient-provider relationship as well as opportunities for improvements in available evidence-based knowledge for patients and providers alike. Ultimately, steps taken to empower patients to seek and gain knowledge about their personal health can have a positive impact on health outcomes. MIND THE GAP 39 References Barbosa, S., de S, R., & Coca Velarde, L. (2013). Diastasis of rectus abdominis in the immediate puerperium: Correlation between imaging diagnosis and clinical examination. Archives of Gynecology and Obstetrics, 288(2), 299-303. doi:10.1007/s00404-013-2725z Belogolovsky, I., Katzman, W., Chirstopherson, N., Rivera, M., & Allen, D. D. (2015). The effectiveness of exercise in treatment of pregnancy-related lumbar and pelvic girdle pain: A meta-analysis and evidence-based review. Journal of Women's Health Physical Therapy, 39(2), 53-64. Retrieved from http://dx.doi.org/10.1097/JWH.0000000000000032 Benjamin, D. R., Frawley, H. C., Shields, N., van de Water, A. T., & Taylor, N. F. (2019). Relationship between diastasis of the rectus abdominis muscle (DRAM) and musculoskeletal dysfunctions, pain and quality of life: A systematic review. Physiotherapy, 105(1), 24-34. doi:10.1016/j.physio.2018.07.002 Benjamin, D. R., van de Water, A. T., & Peiris, C. L. (2014). Effects of exercise on diastasis of the rectus abdominis muscle in the antenatal and postnatal periods: A systematic review. Physiotherapy, 100(1), 1-8. doi:10.1016/j.physio.2013.08.005 B, K., Hilde, G., Tennfjord, M. K., Sperstad, J. B., & Engh, M. E. (2017). Pelvic floor muscle function, pelvic floor dysfunction and diastasis recti abdominis: Prospective cohort study. Neurourology and Urodynamics, 36(3), 716-721. doi:10.1002/nau.23005 Burton, C., Gonzalez, G., Vaculik, K., Khalil, C., Zektser, Y., Arnold, C., Almario, C. V., Spiegel, B. M. R., Anger, J. T. (2020). Female lower urinary tract symptom prevention MIND THE GAP 40 and treatment strategies on social media: Mixed correlation with evidence. Urology. https://doi.org/10.1016/j.urology.2020.06.056 Castro, E. M., Van Ragenmortel, T., Vanhaecht, K., Sermeus, W., & Van Hecke, A. (2016). Patient empowerment, patient participation and patient-centeredness in hospital care: A concept analysis based on a literature review. Patient Education and Counseling, 99(12), 1923-1939. doi:10.1016/j.pec.2016.07.026 Chiarello, C. M. (2017). Pregnancy-related pelvic girdle pain and diastasis rectus abdominis. Research Report, 41(1) doi:10.1097/JWH.0000000000000068 Coldron, Y., Stokes, M. J., Newham, D. J., & Cook, K. (2006). Postpartum characteristics of rectus abdominis on ultrasound imaging. Manual Therapy, 13(2), 112-121. doi:10.1016/j.math.2006.10.001 Corbin, J., & Strauss, A. (2008). Basics of qualitative research (3rd ed.). Thousand Oaks, CA: SAGE Publications. Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: SAGE Publications. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Thousand Oaks, CA: SAGE Publications. da Mota, P., Pascoal, A., Carita, A., & B, K. (2014). Prevalence and risk factors of diastasis recti abdominis from late pregnancy to 6 months postpartum, and relationship with lumbo-pelvic pain. Manual Therapy, 20(1), 200-205. doi:10.1016/j.math.2014.09.002 Dall, T. M., Chakrabarti, R., Storm, M. V., Elwell, E. C., & Rayburn, W. F. (2013). Estimated demand for women's health services by 2020. Journal of Women's Health, 22(7), 643648. doi:10.1089/jwh.2012.4119 MIND THE GAP 41 Daniels, H., Erickson, S. M., Bornstein, S. S. (2018). Womens health policy in the United States: An American college of physicians position paper. Annals of Internal Medicine. https://doi.org/10.7326/M17-3344 Dierckx de Casterl, B., Verhaeghe, S. T., Kars, M. C., Coolbrandt, A., Stevens, M., Stubbe, M., . . . Grypdonck, M. (2011). Researching lived experience in health care: Significance for care ethics. Nursing Ethics, 18(2), 232-242. doi:10.1177/0969733010389253 Doshani, Pitchforth, Mayne, & Tincello. (2009). The value of qualitative research in urogynaecology. BJOG: An International Journal of Obstetrics & Gynaecology, 116(1), 3-6. doi:10.1111/j.1471-0528.2008.01924.x Etikan, I., Musa, A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1. doi:10.11648/j.ajtas.20160501.11 Fahey, J. O., & Shenassa, E. (2013). Understanding and meeting the needs of women in the postpartum period: The perinatal maternal health promotion model. Journal of Midwifery & Women's Health, 58(6), 613-621. doi:10.1111/jmwh.12139 Freeman, R. (2013). Can we prevent childbirthrelated pelvic floor dysfunction? BJOG: An International Journal of Obstetrics & Gynaecology, 120(2), 137-140. doi:10.1111/14710528.12092 Funnell, M. M. (2016). Patient empowerment: What does it really mean? Patient Education and Counseling, 99, 1921-1922. http://dx.doi.org/10.1016/j.pec.2016.10.010 Geynisman, J. M., Tauble, D., & Asfaw, T. S. (2018). Is something missing from antenatal education? A survey of pregnant womens knowledge of pelvic floor disorders. Female MIND THE GAP 42 Pelvic Medicine & Reconstructive Surgery. 24(6), 440-443. doi:10.1097/SPV.00000000000000465 Gustavsson, C., & Eriksson-Crommert, M. (2020). Physiotherapists and midwives views of increased interrecti abdominis distance and its management in women after childbirth. BMC Womens Health. 20(37), 1-9. https://doi.org/10.1186/s12905-020-0090709 Halcomb, E. J., & Davidson, P. M. (2006). Is verbatim transcription of interview data always necessary? Applied Nursing Research, 19(1), 38-42. doi:10.1016/j.apnr.2005.06.001 Henderson, R., & Rheault, W. (2004). Appraising and incorporating qualitative research in evidence-based practice. Journal of Physical Therapy Education, 18(3), 35-40. doi:10.1097/00001416-200410000-00005 Hickey, F., Finch, J., & Khanna, A. (2011). A systematic review on the outcomes of correction of diastasis of the recti. Hernia, 15(6), 607-614. doi:10.1007/s10029-011-0839-4 Jacobs, W., Amuta, A. O., & Jeon, K. C. (2017). Health information seeking in the digital age: An analysis of health information seeking behavior among US adults. Cogent Social Sciences. 3(1), 1-11. http://dx.doi.org/10.1080/23311886.2017.1302785 Jin, J., Sklar, G. E., Oh, V. M. S., & Li, S. C. (2008). Factors affecting therapeutic compliance: A review from the patient's perspective. Retrieved from http://scholarbank.nus.edu.sg/handle/10635/106652 Kamel, D. M., & Yousif, A. M. (2017). Neuromuscular electrical stimulation and strength recovery of postnatal diastasis recti abdominis muscles. Annals of Rehabilitation Medicine, 41(3), 465-474. doi:10.5535/arm.2017.41.3.465 Keeler, J., Albrecht, M., Eberhardt, L., Horn, L., Donnelly, C., & Lowe, D. (2012). Diastasis recti abdominis: A survey of women's health specialists for current physical therapy MIND THE GAP 43 clinical practice for postpartum women. Journal of Womens Health Physical Therapy, 36(3), 131-142. doi:10.1097/JWH.0b013e318276f35f Keshwani, N., Mathur, S., & McLean, L. (2018). Relationship between interrectus distance and symptom severity in women with diastasis recti abdominis in the early postpartum period. Physical Therapy, 98(3), 182-190. doi:10.1093/ptj/pzx117 Lacy, J. W., & Stark, C. E. L. (2013). The neuroscience of memory: Implications for the courtroom. Nature Reviews. Neuroscience, 14(9), 649-658. doi:10.1038/nrn3563 Lagan, B. M., Sinclair, M., & Kernohan, W. G. (2010). Internet use in pregnancy informs womens decision making; A web-based survey. Birth. 37(2), 106-115. https://doi.org/10.1111/j.1523-536X.2010.00390.x Lawson, S., & Sacks, A. (2018). Pelvic floor physical therapy and women's health promotion. Journal of Midwifery & Women's Health, 63(4), 410-417. doi:10.1111/jmwh.12736 Litos, K. (2014). Progressive therapeutic exercise program for successful treatment of postpartum women with a severe diastasis recti abdominis. Journal of Women's Health Physical Therapy, 38(2), 58-73. doi:10.1097/JWH.0000000000000013 Lupton, D. & Maslen, S. (2019). How women use digital technologies for health: Qualitative interview and focus group study. Journal of Medical Internet Research. 21(1). Retrieved from https://www.jmir.org/2019/1 Marshall, C., & Rossman, G. B. (2011). Designing qualitative research (5th ed.). Thousand Oaks, CA: SAGE Publications. Merriam, S. B. (2002). Qualitative research in practice: Examples for discussion and analysis. In S. B. Merriam (Ed.), Basic interpretive qualitative research (pp. 37-39). San Francisco, CA: Jossey-Bass. MIND THE GAP 44 Mommers, E., Ponten, J., Al Omar, A., de Vries Reilingh, T., Bouvy, N., & Nienhuijs, S. (2017). The general surgeon's perspective of rectus diastasis. A systematic review of treatment options. Surgical Endoscopy, 31(12), 4934-4949. doi:10.1007/s00464-017-5607-9 Morse, J. (2000). Determining sample size. Qualitative Health Research, 10(1), 3-5. doi: 10.1177/104973200129118183 Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), 13-22. doi:10.1177/160940690200100202 Naidu, A. (2009). Factors affecting patient satisfaction and healthcare quality. International Journal of Health Care Quality Assurance, 22(4), 366-381. doi:10.1108/09526860910964834 Nayir, T., Uskun, E., Yrekli, M. V., Devran, H., elik, A., & Okyay, R. A. (2016). Does body image affect quality of life?: A population based study. PloS One, 11(9), e0163290. doi:10.1371/journal.pone.0163290 Oh, H. J. & Lee, B. (2012). The effects of computer-mediated social support in online communities on patient empowerment and doctor-patient communication. Health Communication. 27, 30-41. doi:10.108-/10410236.2011.567449 Opala-Berdzik, A., & Dbrowski, S. (2009). Physiotherapy in diastasis of the rectus muscles of abdomen in women during pregnancy and postpartum. Physiotherapy, 17(4), 67. doi:10.2478/v10109-010-0039-2 Parker, M. A., Millar, A. L., & Dugan, S. A. (2008). Diastasis rectus abdominis and lumbopelvic pain and Dysfunction: Are they related? Journal of Women's Health Physical Therapy, 32(1), 15-22. MIND THE GAP 45 Ponmathi, P., Jayalakshmi, T., & Sivakumar, V. P. R. (2016). Prevalence of diastasis recti among postpartum women-an observational study. Indian Journal of Physiotherapy and Occupational Therapy - an International Journal, 10(4), 125. doi:10.5958/09735674.2016.00133.7 Rayburn, W., Klagholz, J., Murray-Krezan, C., Dowell, L., & Strunk, A. (2012). Distribution of american congress of obstetricians and gynecologists fellows and junior fellows in practice in the united states. Obstetrics & Gynecology, 119(5), 1017-1022. doi:10.1097/AOG.0b013e31824cfe50 Romano, M., Cacciatore, A., Giordano, R., La Rosa, B. (2010). Postpartum period: Three distinct but continuous phases. Journal of Prenatal Medicine. 4(2), 22-25. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3279173/pdf/prenatal-04-0022.pdf Rowlands, I. J., Loxton, D., Dobson, A., Mishra, G. D. (2015). Seeking health information online: Associations with young Australian womens physical, mental, and reproductive health. Journal of Medical Internet Research. 17(5). doi:10.2196/jmir.4048 Ryan-Nicholls, K. D., & Will, C. I. (2009). Rigour in qualitative research: Mechanisms for control. Nurse Researcher, 16(3), 70-85. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/19425402 Saldana, J. (2016). The coding manual for qualitative researchers (3rd ed.). Los Angeles, CA: SAGE Publications. Sperstad, J. B., Tennfjord, M. K., Hilde, G., Ellstrm-Engh, M., & B, K. (2016). Diastasis recti abdominis during pregnancy and 12 months after childbirth: Prevalence, risk factors and report of lumbopelvic pain. British Journal of Sports Medicine, 50(17), 1092-1096. doi:10.1136/bjsports-2016-096065 MIND THE GAP 46 Spitznagle, T., Leong, F., & Van Dillen, L. (2007). Prevalence of diastasis recti abdominis in a urogynecological patient population. International Urogynecology Journal, 18(3), 321328. doi:10.1007/s00192-006-0143-5 Stedman, T. L. (2006). Stedman's medical dictionary (28. ed. ed.). Philadelphia, PA: Lippincott Williams & Wilkins. Stenberg, K., Axelson, H., Sheehan, P., Anderson, I., Guelmezoglu, A. M., Temmerman, M., . . . Bustreo, F. (2014). Advancing social and economic development by investing in women's and children's health: A new global investment framework. Retrieved from http://researchonline.lshtm.ac.uk/1726191/ Tidy, C. (2019). Gravidity and Parity Definitions: Implications in Risk Assessment. Patient. https://patient.info/doctor/gravidity-and-parity-definitions-and-their-implications-in-riskassessment Tully, K. P., Stuebe, A. M., Verbiest, S. B. (2017). The fourth trimester: a critical transion period with unmet maternal health needs. American Journal of Obstetrics & Gynecology. http://dx.doi.org/10.1016/j.ajog.2017.03.032 Turan, V., Colluoglu, C., Turkyilmaz, E., & Korucuoglu, U. (2011). Prevalence of diastasis recti abdominis in the population of young multiparous adults in turkey. Ginekologia Polska, 82(11), 817. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/22384613 van de Water, A., & Benjamin, D. R. (2016). Measurement methods to assess diastasis of the rectus abdominis muscle (DRAM): A systematic review of their measurement properties and meta-analytic reliability generalisation. Manual Therapy, 21, 41-53. doi:10.1016/j.math.2015.09.013 MIND THE GAP Verbiest, S., Tully, K., Simpson, M., & Stuebe, A. (2018). Elevating mothers voices: Recommendations for improved patient-centered postpartum. Journal of Behavioral Medicine. 41, 577-590. https://doi.org/10.1007/s10865-018-9961-4 Yeh, M., Wu, S., & Tung, T. (2018). The relation between patient education, patient empowerment and patient satisfaction: A cross-sectional-comparison study. Applied Nursing Research, 39, 11-17. doi:10.1016/j.apnr.2017.10.008 47 MIND THE GAP 48 Participant Characteristics Table 1 Participant Characteristics Pseudonym MD/Self Age Occupation Gravida Para Total # DRA Dx of # of Type of children/birth delivery children Sarah MD 30 Physical Therapist 2 2 2 Singles Natural x2 Myrna Self 30 Administration 3 2 2 Singles Cesarean x2 Linda MD 34 Engineer 1 1 1 Single Cesarean Brit MD 30 Stay at Home Mom 2 2 3 1- Single 1- Twins 1Natural 1Cesarean Sharon MD 31 Design 2 2 2 Singles Jamie Self 27 Photographer 3 2 2 Singles Rebecca MD 34 2 1 1 Single Connie MD 32 2 2 2 Singles Annalise MD 28 2 2 2 Singles Tara Self 26 3 3 3 Singles Jennifer MD 38 4 3 3 Singles Rheema Self 38 2 2 2 Singles Lisa Self 29 Physical Therapist Pediatric Nurse Orthodontic Assistant Stay at Home Mom Physical Therapist Physical Therapist Teacher 1Natural 1Cesarean Natural x2 Natural 2 2 2 Singles Natural x2 Natural x2 Natural x3 Cesarean x3 Cesarean x2 Natural x2 MIND THE GAP 49 Appendix A Recruitment Flyer Mind the GAP: Knowledge of Diastasis Recti Abdominis in Postpartum Women The purpose of this study is to understand the experience women have with separation of the stomach muscles during pregnancy. This separation may not heal after childbirth. This condition is known as Diastasis Recti Abdominis (DRA). Participants needed: Female Given birth 8 weeks to 3 years ago Willing to take part in an interview Speak and understand English Diagnosis of DRA or have symptoms of DRA that can include: o separation along the middle of the stomach muscles o weakness of the stomach muscles o low back pain o unable to control bladder or bowels o pelvic pain or organ prolapse Risks and Benefits of the study: This study has little to no risk to the participant. Participation could help improve medical care for women with DRA by providing information about their experiences. Information collected from this study could improve education about DRA and access to treatment options for women. What will you need to do? This study will involve an audio-recorded interview between the researcher and each person lasting up to 60 minutes. Questions during the interview will ask about your experience with symptoms, diagnosis, education, management and treatment of DRA. Interviews will be in-person but can be over the phone or computer if needed. Participants will review collected information to make sure it represent their experiences, which could take up to 60 minutes. If you have questions or would like to join this study, please contact Jennifer Wiley PT, DPT wileyjj@uindy.edu or 770-540-4321 MIND THE GAP 50 This study is a Doctoral Project for the completion of the Doctor of Health Sciences degree at the University of Indianapolis This study has been approved by the Philadelphia College of Osteopathic Medicine Institutional Review Board (Protocol #H20-002X). For further information on the rights of research subjects, please contact the Research Compliance Specialist at 215-871-6782. MIND THE GAP 51 Appendix B Study Information Sheet Mind the GAP: Knowledge of Diastasis Recti Abdominis in Postpartum Women I would like to invite you to take part in a research study. Please take time to read the following information carefully. Ask questions if anything you read is not clear or if you would like more information. Take time to decide if you want to join this study. WHO I AM AND WHAT THIS STUDY IS ABOUT My name is Jennifer Wiley; I am a student in the Doctor of Health Sciences (DHSc) program at the University of Indianapolis. I am also a physical therapist and assistant professor in the DPT program at PCOM Georgia. This study will serve as my doctoral project for my DHSc degree. The purpose of this study is to understand the experience women have with separation of the abdominal musculature during pregnancy. This separation may not heal following childbirth. This condition is known as Diastasis Recti Abdominis (DRA). WHAT WILL TAKING PART INVOLVE? This study will involve an audio-recorded interview between the primary investigator and each person. Questions during the interview will ask about your experience with symptoms, diagnosis, education, management and treatment of DRA. Interviews will be in-person but can be over the phone or computer if needed. Recordings of each interview will be turned into written text for review. Specific words and quotes from the interviews will be picked out. Each participant will review those words and quotes to make sure they represent their experiences. WHY HAVE YOU BEEN INVITED TO TAKE PART? To participate in this study you must be an English speaking female who gave birth 8 weeks to 3 years ago. You must have experienced symptoms of or medically diagnosed with DRA and willing to speak about your experiences. DO YOU HAVE TO TAKE PART? MIND THE GAP 52 Participation is completely voluntary. Each person has the right to stop participation at any point and leave the study with no consequence. Participants may also ask that parts of the interview not be used. WHAT ARE THE POSSIBLE RISKS AND BENEFITS OF TAKING PART? This study poses little to no risk to the participant. Participation could help improve medical care for women with DRA by providing information about their experiences. Information collected from this study could improve education about DRA and access to treatment options for women. WILL YOUR INFORMATION BE SAFE? In order to keep all information private, each person will be assigned a random (ID) number. Information collected will be kept on a password-protected computer or in a locked storage cabinet. Names, email addresses, and phone numbers will be kept separate in a locked cabinet. After the study, the recorded interviews will be destroyed and all written paperwork will be kept for five years. All participants can ask to access their information at any time. WHAT WILL HAPPEN TO THE RESULTS OF THE STUDY? The results of this study will be used to complete the DHSc degree at the University of Indianapolis. Results may also be used for future presentations that could include conferences, publications, and use in teaching by the primay investigator. WHO SHOULD YOU CONTACT FOR FURTHER INFORMATION? Primary Investigator: Jennifer Wiley PT, DPT PCOM Georgia University of Indianapolis DHSc Student wileyjj@uindy.edu 770-540-4321 Co-Investigators: Laura Santurri PhD, MPH, CPH Project Chair Committee Assistant Professor University of Indianapolis Ruth Maher PT, PhD, DPT Project Content Expert Professor PCOM Georgia Lisa Borrero PhD Doctoral Project Analysis Expert Assistant Professor University of Indianapolis This study has been approved by the Philadelphia College of Osteopathic Medicine Institutional Review Board (Protocol #H20-002X). For further information on the rights of research subjects, please contact the Research Compliance Specialist at 215-871-6782. MIND THE GAP 53 Appendix C Mind the GAP: Knowledge of Diastasis Recti Abdominis in Postpartum Women Demographic Sheet ID #_________________ Date___________________ Age____ Ethnicity________________ Identified Gender_____________ Education Level _______________________ Occupation____________________ Past Medical History Medical Conditions _____________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Surgical History________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Medications ___________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Pregnancy History Number of Pregnancies________________ Number of Births _________________ Number of Children __________________ Dates of Pregnancies and Births ___________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Medical Complications During Pregnancies and During or After Births ____________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Functional Limitations or Concerns During or After Pregnancies _________________________ ______________________________________________________________________________ ______________________________________________________________________________ MIND THE GAP 54 Appendix D Semi-Structured Interview Guide 1. Tell me about yourself and your pregnancy history. a. Can you tell me more about the symptoms you were experiencing following pregnancy? 2. I know you have been diagnosed with or have had symptoms of diastasis recti abdominis; can you tell me what that experience has been like for you? a. Can you tell me more about that experience and how it has made you feel? 3. Can you tell me what your experience was like during your interactions with health care providers regarding your symptoms or diagnosis of DRA? a. To what extent did you seek medical advice about symptoms you were having? b. How would you change your role or the interaction with health care providers in your experience with DRA? 4. Describe for me the sources of education you sought regarding DRA. a. Can you tell me what was most helpful? Least helpful? 5. Can you describe the education that you received regarding DRA before you experienced symptoms or were diagnosed? a. What was the source of the education? b. How much information were you provided? c. How do you feel about the quality of the education that you received? d. How well did this information prepare you for the diagnosis? e. What would you have changed? 6. Can you describe the education that you received regarding DRA after you experienced symptoms or were diagnosed? MIND THE GAP 55 a. What was the source of the education? b. How much information were you provided? c. How do you feel about the quality of the education that you received? d. How well did this information prepare you for the diagnosis? e. What would you have changed? 7. Can you explain your experience following having symptoms of or being diagnosed with DRA? a. How has this diagnosis affected your outlook on your life? 8. Can you tell me your experience with any the treatment or management options for DRA? a. Can you discuss the forms of support you are currently receiving or have received? b. What types of challenges or barriers have you encountered regarding your DRA diagnosis? 9. If you could speak to a pregnant woman what advice would you give them regarding DRA? a. What information would you like each pregnant woman to receive about DRA? 10. Is there anything else you would like to tell me about your experience with DRA?  ...

... RUNNING-RELATED INJURIES IN ULTRARUNNERS Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Joshua D. White, PT, MS Copyright May 31st, 2019 By: Joshua D. White, PT, MS All rights reserved Approved by: Laura Santurri, PhD, MPH, CPH Committee Chair Lisa Borrero, PhD Committee Member Mark F. Reinking, PT, PhD, SCS, ATC Committee Member Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis Running head: RUNNING-RELATED INJURIES IN ULTRARUNNERS Prevention and Management of Running-Related Injuries in Ultrarunners Josh White University of Indianapolis 1 RUNNING-RELATED INJURIES IN ULTRARUNNERS 2 Abstract Ultrarunning, running beyond marathon distances, has grown significantly in popularity over the past decade. Given the high prevalence of running-related injuries and the multifactorial nature of these injuries, the process whereby ultrarunners seek to achieve consistent and injury-free training and to complete running events greater than 42.2 km (26.2 miles) is not well understood. The purpose of this study was to describe the prevention and management strategies used by ultrarunners for running-related injuries. A qualitative, grounded theory approach, involving 37 semi-structured in-depth interviews with 28 ultrarunners and field observation at two different ultramarathon events, resulted in three themes, listen to your body, back on track, and mental toughness. Study participants were at least 18 years of age and had completed at least one ultramarathon within the last three years. Rigid adherence to high-volume training programs and over-scheduling ultramarathon events were considered to be contributory to overuse running-related injuries, whereas training activity guided by perceived exertion and focusing on a particular goal race for the calendar year were described as effective injury prevention strategies. Management strategies included problem-solving to determine appropriate levels of running activity as well as engaging in fitness alternatives perceived to boost the physical and mental health of ultrarunners during time off running. Study participants utilized a me against me strategy for challenges associated with obligatory training tendencies and pressures associated with online performance tracking tools. Positive thinking strategies were deemed useful for ultrarunning participation; however, these strategies led to mixed results for the management of running-related injuries. Keywords: Grounded theory, ultrarunner, running-related injury RUNNING-RELATED INJURIES IN ULTRARUNNERS 3 Table of Contents Title Page .........................................................................................................................................1 Abstract ............................................................................................................................................2 Chapter 1: Introduction ....................................................................................................................5 Background ..........................................................................................................................5 Problem Statement ...............................................................................................................6 Purpose of the Study ............................................................................................................6 Significance of the Study .....................................................................................................7 Chapter 2: Background and Significance ........................................................................................7 The Sport of Ultrarunning....................................................................................................8 Running-Related Injuries .....................................................................................................9 Goal Orientation and Motivating Factors ..........................................................................10 Training and Performance Strategy ...................................................................................11 Nutrition, Hydration, and Sleep Habits..............................................................................15 Running Form and Shoe Wear...........................................................................................17 Cognitive Strategies and Coping Skills .............................................................................17 Mental Toughness and Pain Perception .............................................................................19 Utilization of Coaching and Medical Professional Services..............................................22 Prevention Strategies for Running-related Injuries............................................................22 Intervention Strategies for Running-related Injuries .........................................................23 Ultrarunning Community ...................................................................................................24 Chapter 3: Method .........................................................................................................................26 Study Design ......................................................................................................................26 RUNNING-RELATED INJURIES IN ULTRARUNNERS 4 Participants .........................................................................................................................26 Procedures ..........................................................................................................................27 Data Management and Analysis ........................................................................................33 Rigor and Trustworthiness .................................................................................................37 Chapter 4: Results ..........................................................................................................................40 Chapter 5: Discussion and Conclusion ..........................................................................................81 Table 1 Participant Characteristics and Ultrarunning Experience ...............................................114 Table 2 Theme Prevention Strategies for Running-Related Injuries ........................................115 Table 3 Theme Management Strategies for Running-Related Injuries ....................................118 Table 4 Theme Ultrarunning Participation ...............................................................................119 Figure 1 Grounded Theory Model ...............................................................................................120 Appendices ...................................................................................................................................123 Appendix A Informational Flyer .....................................................................................123 Appendix B Email Recruitment Document .....................................................................124 Appendix C Participant Information Survey ...................................................................125 Appendix D Interview Guide ...........................................................................................126 Appendix E Interview Guide for Events ..........................................................................129 RUNNING-RELATED INJURIES IN ULTRARUNNERS 5 Prevention and Management of Running-Related Injuries in Ultrarunners Ultramarathons are footraces longer than the marathon distance of 42.2 km (26.2 miles) and often involve running on trails through mountainous terrain in excess of 100 km (62.1 miles) (Jaeschke, Sachs, & Dieffenbach, 2016). Ultrarunners are presented with difficulties related to energy intake, hydration, digestion, variable terrain, unpredictable weather, intense fatigue, sleep deprivation, and visual disturbances (Holt, Homan, Youngoh, & Klein, 2014). Nevertheless, ultrarunning as a sport has grown in popularity since the 1980s (Knechtle, 2012) and has experienced a recent surge in participation with a greater than four-fold increase in the number of ultramarathon event finishes recorded between 2007 and 2017 in the United States ("DUV Marathon Statistics," 2018). A distance runner has been defined as an individual who runs 5 km per training session and/or race event (van Gent et al., 2007). Unfortunately, the annual incidence of running-related injuries varies from 20-80% for distance runners according to a systematic review performed by van Gent et al. (2007). This large variation is due to differences in the definition of injury, study populations, and time period follow-up. In a longitudinal study involving 1,212 active ultramarathon runners, the majority of participants (65%) had a running-related injury resulting in a significant loss in training days (median of 14 days) within the previous year (Hoffman, Krishnan, & Lucia, 2014). The most common injuries affecting ultrarunners are knee injuries (including iliotibial band issues), back injuries, calf strains, hamstring strains, Achilles tendinopathies, and ankle sprains, as well as plantar fasciitis (Hoffman & Fogard, 2011). Stress fractures are also a commonly reported injury for ultrarunners but are more common in the foot than the lower leg, which is the site of most stress fractures in distance runners (Hoffman, 2016; Snyder, Koester, & Dunn, 2006). RUNNING-RELATED INJURIES IN ULTRARUNNERS 6 Problem Statement To date, the majority of research on running-related injuries in distance runners has focused on risk factors, while most literature on ultrarunning has examined the physiology of these endurance athletes (Balducci, Clmenon, Trama, Blache, & Hautier, 2017; Giandolini et al., 2016; van der Worp et al., 2015). A few qualitative studies have described the event experiences of ultra-runners and mentioned management efforts for injuries sustained during particular race events (Holt et al., 2014; Jaeschke et al., 2016; Simpson, Post, Young, & Jensen, 2014). However, there were no studies that described the process whereby ultrarunners approach running-related injuries. Given the rapid growth and unique challenges presented in the sport of ultrarunning, as well as the high incidence of injury associated with distance running, this study explored how ultrarunners implement prevention strategies aimed at maintaining injury-free training, as well as management strategies focused on existing running-related injuries. Purpose of the Study This study utilized a grounded theory approach to collect and analyze qualitative data and to describe how ultrarunners approach prevention and management strategies for running-related injuries. Data collected through interviews of ultrarunners and observation of ultrarunning training and competition provided insights into how different physical and mental strategies are implemented by ultrarunners. Additionally, the study described the resources ultrarunners utilize in the prevention and management of running-related injuries. Research Questions How do ultrarunners approach prevention and management strategies for running-related injuries? RUNNING-RELATED INJURIES IN ULTRARUNNERS o What are the facilitators and barriers in the prevention and management of runningrelated injuries in ultrarunners? o What resources serve to assist ultrarunners in the prevention and management of running-related injuries? Significance of the Study The research findings and conceptual model presented, can be used to provide coaches and healthcare professionals with alternative strategies in the prevention or management of running-related injuries in ultrarunners. Theory development provides insight into the physical and mental aspects of ultrarunning and can be used to guide future research aimed at identifying risk factors for running-related injuries in ultrarunners. Definition of Terms Running-related injury: Running-related (training or competition) musculoskeletal pain that causes a restriction or stoppage of running (including distance, speed, or duration) for at least seven days or three consecutive scheduled training sessions, or that requires the runner to consult a physician or other health professional (Yamato, Saragiotto, & Lopes, 2015). Background and Significance The traditional grounded theorist position advocates for delaying the literature review in the research study process until after completing an analysis (Glaser, Strauss, & Strutzel, 1968). Conversely, Charmaz (2014) recommends engaging with the literature critically and comparatively early in research project development. The latter position encourages an ongoing interaction with evidence which can serve to highlight developing theoretical categories (Charmaz, 2014). The focus of this literature review was to explore the literature related to the 7 RUNNING-RELATED INJURIES IN ULTRARUNNERS 8 sport of ultra-distance running and to give particular attention to the following areas: goal orientation and motivating factors, training and performance strategy, cognitive strategies, coping skills, mental toughness, coaching and medical professionals, and the influence of other runners and supporters. The Sport of Ultrarunning Ultramarathon events involve running farther than the marathon distance of 42.2 km (26.2 miles) (Knechtle, 2012). Race distances are typically 50 km, 50 miles (80 km), 100 km, or 100 miles (161 km) (Hoffman & Krishnan, 2013). Races may also be structured using designated periods of time, such as 12-hour, 24-hour, and multi-day events (Hoffman & Krishnan, 2013). Ultrarunning as a sport has grown in popularity since the 1980s (Knechtle, 2012) and has experienced a recent surge in participation with a greater than four-fold increase in the number of ultramarathon event finishers recorded between 2007 and 2017 in the United States ("DUV Marathon Statistics," 2018). Ultramarathon running continues to represent a relatively small percentage of recreational runners. For example, in 2016 there were less than 94,000 finishers in U.S. ultramarathon events; whereas, over 17 million recreational runners completed a marathon or shorter distance road race event in the same year ("UltraRunning Finishes - U.S.," 2018). The median age for ultrarunners is 41 to 42 years, while the median age of marathon participants is 36 to 37 years (Hoffman, Badowski, Chin, & Stuempfle, 2016; "United States Road Race Trends," 2018). Ultramarathon participation among women has greatly increased and accounts for approximately 30% of ultramarathon participants in North America (Hoffman, 2016; "UltraRunning Finishes - U.S.," 2018). The sport of ultrarunning is characterized by extreme physical and psychological fatigue with overnight running, and exposure to changing weather patterns and variable temperatures, considered to be expected and normal parts of RUNNING-RELATED INJURIES IN ULTRARUNNERS 9 ultrarunning participation (Holt et al., 2014; Jaeschke et al., 2016; Simpson et al., 2014). Nonetheless, a study of 1,212 active ultramarathon runners found that ultrarunners were healthier, utilized fewer outpatient medical visits, and reported fewer missed work or school days due to illness or injury compared to the general population (Hoffman et al., 2014). Running-Related Injuries Yamato et al. (2015) defined a running-related injury as a running-related (training or competition) musculoskeletal pain in the lower limbs that causes a restriction (distance, speed, duration, or training) or stoppage of running for at least 7 days or 3 consecutive scheduled training sessions or that requires the runner to consult a physician or other health professional. Running-related injuries may cause runners to restrict the distance, speed, or duration of their running or to alter their training by changing the running surface, terrain, or periodization strategy (Ferley, Osborn, & Vukovich, 2014; Kenneally, Casado, & Santos-Concejero, 2017; Yamato et al., 2015). The causes of running-related injuries are multifactorial, and strategies for how to remain injury-free for distance runners is a topic that is not well understood (Fuller et al., 2017; Kluitenberg et al., 2016; Kluitenberg, van Middelkoop, Diercks, & van der Worp, 2015; van der Worp et al., 2015). Risk factors for running-related injuries in the sport of ultrarunning have been explored and include training distance, age, experience level, typical running surface, exercise intensity, and use of alternate training activities (Hoffman, 2016). Studies support that 50-60% of ultrarunners will be confronted with running-related injuries in a given year (Hoffman et al., 2014). Injury rates and types of injuries are similar for ultramarathon runners compared with other distance runners, and the types of running-related injuries are also similar (Hoffman et al., 2014; van Gent et al., 2007; Videbaek, Bueno, Nielsen, & Rasmussen, 2015). According to Hoffman et al., 2014, the incidence of exercise-related RUNNING-RELATED INJURIES IN ULTRARUNNERS 10 injuries reported in the previous 12 months among 1,212 active ultramarathon runners were as follows: knee issues (24%); iliotibial band issues (15.8%); calf strain (13.1%); back injuries (12.4%); hamstring strain (11.8%); and, Achilles tendinitis or tear (10.8%). Both medial tibial stress syndrome (MTSS) and chronic exercise-induced compartment syndrome (CECS) are lower leg injuries affecting ultra-endurance athletes (Almekinders & Engle, 2019). Stress fractures account for 3.7% of running-related injuries for ultrarunners (Hoffman et al. 2014). While the lower leg (tibia and fibula) is the most common area for stress fractures in shorter distance runners (Snyder et al., 2006), ultrarunners more often suffer stress fractures affecting the foot (Hoffman, 2016). It has been suggested that this could be related to running surfaces, as it has been reported that half of the training for ultrarunners takes place on trails rather than concrete or asphalt (Hoffman, 2016). Peritendinitis of the extensor tendons of the anterior ankle at the extensor retinaculum or ultramarathoners ankle has been described as an injury that seems to be commonly associated with ultramarathoners (Almekinders & Engle, 2019). Almekinders and Engle (2019) postulated that fear of weight gain leading to decreased performance could lead to dietary restriction potentially harmful to bone health. Goal Orientation and Motivating Factors Ultrarunners identified health, social connection, weight control, and personal goal achievement as reasons for running, as well as psychological reasons, including psychological coping and life meaning (Krouse, Ransdell, Lucas, & Pritchard, 2011). Interestingly, Hoffman and Krouse (2018) also reported that ultrarunners included health as a motivator for participation in running ultramarathons. However, 74.1% of ultrarunners in the same study answered, no when asked whether or not they would discontinue ultramarathon training and participation if they learned with absolute certainty that it was bad for their health (Hoffman & Krouse, 2018). RUNNING-RELATED INJURIES IN ULTRARUNNERS 11 The authors postulated that the response described may be the result of ultrarunners being motivated by achievement and directed by a task-orientated goal style. Roebuck et al. (2018) identified the most important factor motivating ultra-runners to participate in the sport as the opportunity to achieve personal goals including the psychological drive to explore physical and mental limits. An apparent inconsistency regarding health as a motivator raises questions regarding the underlying motivation and goal orientation for these athletes which may affect prevention strategies for ultrarunners. The literature supports the need for professional coaches and health care providers to consider the goal orientation and motivating factors of athletes participating in the sport of ultrarunning (Breslin, Murphy, Kremer, McClean, & Davison, 2014; Gledhill, Forsdyke, & Murray, 2018; Ivarsson et al., 2017). Using the Motivations of Marathoners Scales, results from a survey of 1,349 ultrarunners revealed that that intrinsic motivation is a more important motivating factor than recognition motivation (Hoffman & Krouse, 2018), which could have implications for professional coaches working with ultrarunners. Covassin et al. (2015) identified that an athletes motivation to participate in the management of a sports-related injury in the rehabilitation setting as a critical factor in returning to preinjury levels. Lastly, Buck, Spittler, Reed, and Khodaee (2018) used the Exercise Addiction Inventory to study the psychological attributes of ultrarunners and reported that 20% of respondents screened positive for exercise addiction concerns, which may also play a significant role in how motivational factors related to injury prevention and management of ultrarunners are approached by sports psychologists. Training and Performance Strategy Simpson et al. (2014) identified preparation and strategy as a key theme when describing the experiences of ultrarunners and highlighted the need for rigorous training prior to event RUNNING-RELATED INJURIES IN ULTRARUNNERS 12 preparation. Running distances of more than 40 miles per week are typical for ultrarunners and have been reported in the literature as a risk factor for running-related injuries (Fields, Sykes, Walker, & Jackson, 2010; Hoffman et al., 2014; Knechtle, Rosemann, Zingg, Stiefel, & Rst, 2015; Ristolainen, Kettunen, Waller, Heinonen, & Kujala, 2014; van der Worp et al., 2015; van Gent et al., 2007). Knechtle et al. (2015) reported the weekly training distance for ultrarunners as between 49.2 km (30.6 miles) and 120.8 km (75 miles). Damsted et al. (2019) studied half-marathon runners and identified that significantly more runners were injured when they increased their weekly running distance by greater than 20%. The authors suggested that alterations in running schedules should be considered as a way to minimize injury risk (Damsted et al., 2019). Interestingly, Hoffman et al. (2014) collected survey information on 1,212 ultrarunners and discovered that training distances reported for those who had and had not sustained an injury were similar. This suggests that there may be determinants of injury for distance runners other than weekly mileage run. The same authors reported that the ultrarunners identified themselves as having experienced a running-related injury were younger, less experienced runners who had spent a greater proportion of their exercise time at high intensity (Hoffman et al. 2014). Blood flow restriction training involves resistance training with low loads, in conjunction with an applied occlusion to restrict blood flow in order to increase muscle size and strength in athletic populations (Cook, Kilduff, & Beaven, 2014). The technique has been shown to be safe and effective at increasing muscular strength and endurance in healthy fitness populations and is under study for its use in postinjury scenarios (Loenneke & Pujol, 2009). Paton, Addis, and Taylor (2017) investigated blood flow restriction training and reported that there also appears to be some increase in exercise heart rate with additional enhancements that include physiological RUNNING-RELATED INJURIES IN ULTRARUNNERS 13 and functional measures in comparison to performing the same training without blood flow restriction therapy. Yow, Tennent, Dowd, Loenneke, and Owens (2018) conducted a case study report on a 29-year old active duty soldier who sustained an Achilles tendon rupture and had physical therapy that incorporated blood flow restriction therapy into a standard post-surgical protocol. The authors reported that the individual had a favorable outcome that included a return to running and sports on completion of six weeks of blood flow restriction-assisted therapy. In addition to running distance, pacing has been described as a critical part of ultramarathon event strategy as it relates to training activity and event performance. Different pacing strategies have been described and include positive (fast to slow), negative (slow to fast), even (constant), and variable (terrain-following) (Pruitt & Hill, 2017; Tan, Tan, & Bosch, 2016). Additionally, prescriptive pacing guidelines based on the particular athlete and event have been described (Bossi et al., 2017; Pruitt & Hill, 2017; Tan et al., 2016). Knechtle et al. (2015) identified older runners (40 to 44 years) as able to achieve negative pacing in the last segment of the race, while younger runners (18 to 24 years) had the greatest decrease in running speed with each segment of the race. Given that much of the information on the training and competition of ultrarunners has been gathered through survey instruments, semi-structured interviews and field observation may be useful in providing a more in-depth perspective on ultrarunning activity and running-related injuries. A few qualitative studies focused on the event experiences of ultrarunners and described the psychological benefits of having completed training activity that prepared the athlete for particularly difficult sections faced during event participation (Antonini Philippe, Rochat, Vauthier, & Hauw, 2016; Jaeschke et al., 2016; Simpson et al., 2014). Similarly, Schumacher, Becker, and Wiersma (2016) explored the experiences of channel swimmers who described the RUNNING-RELATED INJURIES IN ULTRARUNNERS 14 benefit of experiencing specific challenges (wind, rough seas, cold, pain, and feeling sick) during training activity as a critical part of their planning and strategy for channel crossings. To date, ultrarunner perceptions of the relationship between training activity, event participation, and strategy related to the prevention and management of running-related injuries is a part of ultrarunning that has not been described in the literature. Overreaching and overtraining. Performance increases in long distance running are achieved through increased training loads interspersed with periods of rest and recovery (Kenneally et al., 2017; Meeusen et al., 2013). Successful training involves increased training loads but also must avoid the combination of excessive training loads plus inadequate recovery (Meeusen et al., 2013). The literature uses the terminology overreaching and overtraining as a continuum delineated by symptomology, rest time required for recovery, and performance outcomes (Kreher & Schwartz, 2012; Meeusen et al., 2013). Overreaching is defined as an accumulation of training load followed by temporary performance decrements, requiring days or weeks for recovery (Kreher & Schwartz, 2012). When followed by an adequate rest period, overreaching can lead to positive adaptation and desired performance outcomes (Kellmann et al., 2018). Kreher and Schwartz (2012) defined overtraining syndrome as a maladapted response to excessive exercise without adequate rest, resulting in perturbations of multiple body systems (neurologic, endocrinologic, immunologic) coupled with mood changes. The cytokine hypothesis is a proposed etiology for overtraining syndrome and is characterized by extreme overreaching combined with injury or overuse to the musculoskeletal system with resultant inflammation (Hohl et al., 2009; Kreher & Schwartz, 2012; Le Meur et al., 2014; Main et al., 2010; Pereira et al., 2014; Smith, 2000). Given the potential health consequences of overtraining, a deeper RUNNING-RELATED INJURIES IN ULTRARUNNERS 15 understanding of ultrarunner perspectives on experiences related to overreaching and overtraining syndrome could provide helpful insights into ultrarunner perceptions related to fatigue and underperformance for ultrarunners. Nutrition, Hydration, and Sleep Habits With some isolated exceptions, much of the literature on ultrarunning related to nutrition and hydration strategies is focused on performance rather than injury prevention or recovery. Pruitt and Hill (2017) developed a model providing running athletes with a prescriptive guideline for optimal pacing and carbohydrate intake to ensure adequate energy levels for ultramarathon event participation. In a separate study, Wardenaar et al. (2018) investigated strategies for optimal fluid intake during a 120 km race. The topic of nutrition and stress fractures may also be significant considerations for ultrarunners and contribute to the prevention and management of running related injuries in ultrarunning. Micklesfield, Hugo, Johnson, Noakes, and Lambert (2007), who studied women runners participating in the ultra- and half-marathons of a particular race event, discovered that bone stress injuries may be associated with high levels of energy expenditure. Nieves et al. (2010) also studied bone density in female runners and performed a two-year prospective cohort study with 125 runners, ages 18 to 26 years. In this study, the authors concluded that in young female runners, low-fat dairy products and the major nutrients in milk (calcium, vitamin D, and protein) were associated with lower stress fracture rate. Lentino, Purvis, Murphy, and Deuster (2013) suggested that sleep deprivation may lead to injuries among the military who are engaging in high levels of demanding physical activity. Martin, Arnal, Hoffman, and Millet (2018) surveyed 636 ultrarunner study participants and reported that 74% had a strategy to manage sleep preceding an ultramarathon, with 54.7% trying to increase their opportunities for sleep. Ultrarunners are presented with unique challenges RUNNING-RELATED INJURIES IN ULTRARUNNERS 16 related to nutrition, hydration and sleep deprivation, particularly during event participation. Therefore, it would be beneficial to gather information on how these topics affect prevention and management of running-related injuries. Running Form and Shoe Wear Other studies have described the specifics of running form and discussed possible implications for running related injuries. Almeida, Davis, and Lopes (2015) reported differences in biomechanical characteristics between foot-strike patterns when running, while Davis, Bowser, and Mullineaux (2016) found that impact-related variables were higher in those with medically diagnosed injuries compared with those who had never been injured in a group of female recreational runners. The effects of ultramarathon event participation on running mechanics has been studied with decreased peak vertical ground reaction force and increased step frequency observed over the course of the race. The authors suggested that the described running pattern may help reduce the deleterious physical effects of ultradistance running (Degache et al., 2016; Giovanelli, Taboga, & Lazzer, 2017). The influence of shoe wear has also been studied and offers distance runners information to consider when adopting prevention and management strategies for running-related injuries (Bertelsen, Jensen, Nielsen, Nielsen, & Rasmussen, 2013; Goss & Gross, 2012; Knapik et al., 2009). In addition to fit and style, shoe manufacturers have presented the concepts of foot type, pronation control, and force of landing during running as factors to consider with regard to shoe selection and running-related injuries. Malisoux et al. (2016) performed a randomized controlled trial on 372 recreational runners and reported that the overall injury risk was lower among the participants who had received motion control shoes for individuals with pronated feet. Minimalist footwear is a type of running shoe characterized by a minimal heel to toe drop and RUNNING-RELATED INJURIES IN ULTRARUNNERS 17 absence of cushioning, while maximalist footwear features an oversized midsole (Esculier, Dubois, Dionne, Leblond, & Roy, 2015; Sinclair, Fau-Goodwin, Richards, & Shore, 2016). Sinclair et al. (2016) reported that the loading rate and tibial acceleration were significantly larger in the minimalist compared to the maximalist and conventional footwear and concluded that minimalist footwear may place runners at increased risk from overuse injury. Davis, Rice, & Wearing (2017) described the benefits of minimalistic shoe wear for running-related injuries and identified a greater tendency to use a mid-foot or forefoot strike in runners wearing minimalistic shoe wear. Cognitive Strategies and Coping Skills Gledhill et al. (2018) reported that psychological interventions, particularly those with a stress reduction focus, are an effective and efficient method for reducing sports injury rates and injury time loss. Similarly, Ivarsson et al. (2017) examined the relationship between psychosocial variables and injury rates, along with the effectiveness of psychological interventions (i.e. stress management, goal setting, and cognitive strategies) aimed at reducing injury occurrence. These authors concluded that negative life-event stress and strong stress had the strongest associations with injury rates. Additionally, the authors found that the participants in the study who received psychosocially-based interventions showed fewer injuries in comparison to the control group (Ivarsson et al., 2017). In many cases, ultra-endurance athletes have found cognitive strategies to be necessary for coping with the extreme challenges of event participation (Holt et al., 2014; Jaeschke et al., 2016; Simpson et al., 2014). In a seminal study of marathon runners, Morgan and Pollock (1977) first delineated between associative strategies (focused on maintaining awareness of physical factors necessary to optimize performance) and dissociative strategies (focused on distracting themselves from sensory information). Hanson and RUNNING-RELATED INJURIES IN ULTRARUNNERS 18 Buckworth (2015) discovered that runners with higher fitness levels used more associative than dissociative strategies overall compared to runners with lower fitness levels. Philippe and Seiler (2005) studied athletes who participated in running, swimming, and cycling and identified differences in the utilization of association and dissociation strategies: males tended to pay close attention to bodily signals, and categories, whereas females preferred to utilize dissociation strategies and shunned sensory inputs. Other mental strategies commonly used by ultrarunners and described in the literature include self-talk, inner dialogue, imagery, and goal setting, as well as attentional focus which includes both associative and dissociative strategies (Holt et al., 2014; Jaeschke et al., 2016; Multhaupt & Beuth, 2018; Simpson et al., 2014). Bhlmayer, Birrer, Rthlin, Faude, and Donath (2017) defined mindfulness as a present-oriented form of mental training and concluded that mindfulness practice strategies be included as complementary mental skills training approach for athletes, particularly in precision sports such as pistol shooting. De Petrillo, Kaufman, Glass, and Arnkoff (2009) investigated the effect of mindfulness meditation on long-distance runners and discovered no improvements in performance, but did report decreased sport-anxiety related worry, and aspects of perfectionism. A systematic review described the use of imagery in rehabilitation for athletic injuries involving concussions, shoulder, knee, and ankle injuries, as well as a mix of sports-related injuries (Multhaupt & Beuth, 2018). Several studies demonstrated the efficacy of imagery in the rehabilitation following injury. Schumacher et al. (2016) explored the coping mechanisms of channel swimmers participating in 12 to 16 hour-long open water swimming from one land mass to another. These endurance athletes described actively reframing negative thoughts into positive ones and identifying pain as a normal part endurance sport participation. The practice of goalsetting in ultrarunning events involves thinking of the race in small sections which allows RUNNING-RELATED INJURIES IN ULTRARUNNERS 19 athletes to focus on completing a specific goal more manageable than the entire course (Hoffman & Fogard, 2011; Holt et al., 2014). Despite the distances run by ultrarunners being greater than that reported by shorterdistance runners, and the fact that running distance has been found to be a risk factor for injury (van der Worp et al., 2015; van Gent et al., 2007), the injury rate was not higher among ultramarathon runners competing at the 161km distance compared with ultrarunners competing in shorter distance ultrarunning events. It has been proposed that the reduced risk relative to the distance run may be a result of an adaptation process in response to stress associated with ultramarathon running, other intrinsic characteristics, or the result of developed cognitive strategies and coping skills (Hoffman et al., 2014; Holt et al., 2014). Slower running speed, increased stride frequency, and lower impact forces, as well as trail rather than paved surfaces, have also been suggested as important in mitigating overuse injuries for ultramarathon runners (Hoffman & Krishnan, 2013; Knechtle, 2012). Mental Toughness and Pain Perception The sport of ultrarunning requires that runners maintain a certain mental toughness in order to properly perceive pain and demonstrate sound decision-making while enduring the suffering associated with long bouts of demanding activity. Ultrarunner perceptions regarding the balance of running and resting as it pertains to potential or existing running-related injuries is not well understood. Crust and Azadi (2010) described mental toughness as a multifaceted construct made up of cognitive, emotional, situational, and behavioral components. Jaeschke et al. (2016) provided an expanded understanding of mental toughness by conducting a qualitative analysis of 408 ultramarathon runners who reported on perceptions related to the challenges associated with ultrarunning. These authors then defined mental toughness as, the ability to RUNNING-RELATED INJURIES IN ULTRARUNNERS 20 persist and utilize mental skills to overcome perceived physical, psychological, emotional, and environmental obstacles in the relentless pursuit of a goal (Jaeschke et al., 2016, p. 64). According to these authors, a critical component of mental toughness as it relates to ultradistance running is the ability to differentiate between training pain and potential-injury pain; this skillset requires athletes to make difficult decisions that potentially involve scaling back on intensity, discontinuing training session, or withdrawing from race events. Jelvegrd, Timpka, Bargoria, Gauffin, and Jacobsson (2016) used a qualitative research design to study elite middle and long-distance runners (ultramarathon runners were excluded) and described the behavior of this group of runners, which included altering their self-management of injury differently depending upon whether an insidious or sudden onset of symptoms occurred. While several qualitative studies have described the event experiences of ultrarunners and alluded to the thought or decision-making processes involved with injury management efforts during race events (Holt et al., 2014; Simpson et al., 2014), there is little understood about the relationship between how injuries develop and how this impacts ultrarunner thought processes. The need for ultrarunners to differentiate between training pain and potential-injury pain has been described as an essential part of the sport (Jaeschke et al., 2016). Hoffman and Fogard (2011) reported that 7.5% of finishers and 15.8% of non-finishers who participated in a 161-km event reported having an ongoing injury prior to event participation. Antonini Philippe et al. (2016) described the experiences of 10 runners who failed to finish ultramarathon race events and received a DNF (Did Not Finish) status. The authors observed six phases preceded the final decision to stop running: feeling pain, assigning meaning to those feelings, adjusting running style, attempting to overcome the problem, other runners influences, and assessing the situation. Pain was included as an early part of the sequence and in some instances included feeling RUNNING-RELATED INJURIES IN ULTRARUNNERS 21 physical pain related to muscle, joint, nerve, and bodily pains, primarily occurring in the lower limbs (Antonini Philippe et al., 2016). The authors of the study concluded that there was a need for further research to investigate the transition from monitoring their bodies without the need for any particular cognitive or management strategy to actively investigating for the source of the disruptor or pain. Antonini Philippe et al. (2016) identified associating feelings about health with fears and doubts about the potential consequences of the pain as occurring prior to the discontinuation of event participation. Freund et al. (2013) identified a greater pain tolerance in a small group of ultramarathon runners who were compared to a control group. Adaptations to the physical demands and stress required by ultramarathon running as well as intrinsic characteristics unique to extreme endurance athletes are possible factors influencing pain perception for ultrarunners (Holt et al., 2014). Alschuler et al. (2019) studied the effect of pain coping strategies and pain related outcomes in 204 entrants to an ultrarunning event. The authors described maladaptive approaches as unhelpful and adaptive approaches as helpful in relation to function and ultrarunning activity. The ultrarunners studied utilized adaptive coping strategies more often than maladaptive strategies. The study findings support the idea that optimizing pain coping strategies may be critical to reducing factors that may interfere with ultramarathon performance (Tan, Teo, Anderson, & Jensen, 2011). In two studies also dealing with pain perception, endurance athletes were interviewed and described positive experiences with adopting an accepting attitude toward pain associated with endurance sport participation (Schumacher et al., 2016; Simpson et al., 2014). Lastly, a study with a repeated-measures design tested 21 ultrarunners by using a standardized device to perform a finger prick at the start and finish of a 100-mile endurance run (Hoffman, Lee, Zhao, & RUNNING-RELATED INJURIES IN ULTRARUNNERS 22 Tsodikov, 2007). A group of race event volunteers served as the control group. In this study, the exercise-induced analgesia effect was present at the end of the 100-mile race but only for the faster runners and was not observed among the slower runners (Hoffman et al., 2007). This suggests that the thought and decision-making process regarding pain perception for ultrarunners may be experienced differently depending upon the ability level of the running athlete, which could have implications for running injury management during race events. Utilization of Coaching and Medical Professional Services One poorly understood aspect of the ultrarunning community is the process whereby ultrarunners decide to seek out coaching and medical professional services for running-related injuries. Krouse, Ransdell, Lucas, and Pritchard (2011) reported that over three quarters of women ultrarunners did not use a coach and cited cost as the most common reason for not seeking out professional coaching services. According to Hoffman et al. (2014), the percentage of 1,212 active ultramarathon runners reported to have visited specific types of health care professionals in the prior 12 months were as follows: massage therapy (48%); chiropractic (22%); physical, occupational, and speech therapy (22%); podiatry (13%); and, acupuncture (7%). Prevention Strategies for Running-related Injuries The literature provides some support for the role that running coaches may play a role in prevention and management strategies for running-related injuries. For example, Hespanhol, van Mechelen, and Verhagen (2018) studied the effectiveness of adding online tailored advice to general advice on the prevention of running-related injuries for 232 trail runners and determined that the intervention group sustained 13% fewer running-related injuries compared with those in the control group after six months of follow-up. Mucha, Caldwell, Schlueter, Walters, and RUNNING-RELATED INJURIES IN ULTRARUNNERS 23 Hassen (2017) investigated the association between hip abduction strength and lower extremity running-related injury in distance runners and concluded that weakness of the hip abductors may be associated with iliotibial band syndrome. A number of studies support the use of exercise routines that include strengthening and proximal control exercises to provide preventive neuromuscular training interventions designed to reduce injury in athletes (Sugimoto, Myer, Foss, & Hewett, 2015; Wouters et al., 2012; Yoo et al., 2010). Kozinc and arabon (2017) performed a systematic review of the literature regarding movement therapy interventions and training-modification interventions for running-related injury prevention. These authors identified only seven articles that met the criteria of a randomized controlled trial or prospective study and found no evidence to support the preventative effects of movement therapy or training modifications. Yeung and Yeung (2001) also performed a systematic review of interventions to prevent lower limb soft tissue running injuries and identified exposure to a high training load including duration, frequency, and running distance increases the risk of injury. These authors suggested that modification of the training schedule can reduce the incidence of injury. The effectiveness of stretching exercises and of insoles in the prevention of lower extremity soft tissue injuries caused by running were unclear, while some evidence suggested that wearing a knee brace with patellar support may be effective in the prevention of anterior knee pain caused by running. Intervention Strategies for Running-related Injuries It is beyond the scope of this study to provide an exhaustive search of the efficacy for intervention strategies provided by different medical professionals treating running-related injuries. Overall, there is a need for rigorous randomized clinical trials with larger sample sizes to establish clinical evidence for many of the treatment interventions sought out by ultrarunners. RUNNING-RELATED INJURIES IN ULTRARUNNERS 24 For example, Kim et al. (2014) performed a systematic review and reported that the effects of acupuncture for the treatment of acute ankle sprains does not provide reliable support for either the effectiveness of acupuncture treatments, as a stand-alone treatment, in combination with other nonsurgical interventions or in comparison with other nonsurgical interventions. Mason, Dickens, and Vail (2012) summarized that most techniques for rehabilitation of hamstring injuries have not been assessed using randomized trials. These authors did report evidence to suggest that exercise to correct movement dysfunction could reduce time to return to full activity, but found little to no evidence to suggest that time to recover for athletes can be reduced with an increased daily frequency of hamstring stretching exercises. The Orthopaedic Section of the American Physical Therapy Association (APTA) has established guidelines for orthopaedic physical therapy management of certain musculoskeletal impairments that are running-related injuries. For example, Martin et al. (2014) provided clinical practice guidelines for nonarthritic heel pain, which is a condition that commonly affects ultrarunners (Hoffman et al. (2014). Heapy et al. (2018) investigated post-race recovery for 56 ultrarunners by performing a randomized controlled trial involving two separate interventions and a control group. In this study, a specific manual therapy protocol and intermittent pneumatic compression treatment were compared with supine rest. It was reported that both intervention groups were effective for decreasing muscle pain and soreness following an ultramarathon race event (Heapy et al., 2018). Ultrarunning Community The ultra-distance running community includes the running athletes, volunteers, and support crews which often include friends and family members. It is described in the literature as providing a sense of community and an important source of social support for those who participate in ultrarunning (Holt et al., 2014). Similarly, Simpson et al. (2014) also identified a RUNNING-RELATED INJURIES IN ULTRARUNNERS 25 theme of community with a shared sense of support, common bonds, and camaraderie amongst ultrarunners. A sense of exclusivity and respect for fellow race participants regardless of ability level was noted by multiple authors (Antonini Philippe, Rochat, Vauthier, & Hauw, 2016; Holt et al., 2014; Simpson et al., 2014). Online community forums may also function as a valuable resource for ultrarunners and have implications for considering the role of the ultra-community and the prevention and management of running-related injuries. Rochat, Hauw, Gr, and Seifert (2018) performed an inductive content analysis and concluded that successions of questions and solutions are meaningful not only for building, enriching, and adjusting their trail running activities, but also for coping with constraints related to running injuries. In this study and others like it, the authors demonstrated information sharing via a particular online trail running focused platform (in this case specific to trail running) related to topics such as running injury, suggestions for exercises programs, recommended referral sources, and suggestions for alterations in training schedule (Kimmerle, Gerbing, Thiel, & Cress, 2012; Rochat et al., 2018). Summary The vast majority of research published on running-related injuries has focused on risk factors, while the literature on ultrarunning has focused on the physiology of ultrarunners. While a number of qualitative studies have described the event experiences of ultrarunners in general terms, this is the first article published specifically on ultrarunner experiences as it relates to strategies for the prevention and management of running-related injuries. Given the limited research on the topic of utilization strategies for injuries and the uniquely intense time and volume requirements in the sport of ultrarunning, a deeper understanding of how different physical and mental strategies are implemented by ultrarunners was needed. The literature does support the notion that mental toughness, pain perception, cognitive skills, coping strategies, RUNNING-RELATED INJURIES IN ULTRARUNNERS 26 motivating factors, and the nature of the ultrarunning community are relevant areas of inquiry for a qualitative research study with an inductive and open-ended approach consistent with grounded theory. Method Study Design This study provided a relevant model and grounded theory to address a dearth of literature regarding behavior related to the prevention and management of running-related injuries in ultrarunners. The grounded theory methodology allowed for an inductive process to occur, whereby data collection and analysis occurred in parallel. In this study, the analysis of data collected via semi-structured, in-depth interviews and field observations were applied to a grounded theory construct aimed at describing the behavior of ultrarunners with regard to prevention and management strategies for running-related injuries. Institutional Review Board approval was sought and obtained from the University of Indianapolis. Participants Initially, sampling strategies targeted the recruitment of runners in such a way that the heterogeneity of the ultrarunner population was represented (Padgett, 2012). Consistent with the transition for sampling strategy described by Claire, Donna, Ratchneewan, and Thomas (2007), sampling began with selective sampling and transitioned to theoretical sampling as concepts began to emerge. The number of ultrarunners participating in the study, as well as the age and gender of the participants, experience-level of the runner (including specifics on ultramarathon event participation), and a history of exercise-related medical issues was reported. Inclusion criteria required participants to be 18 years of age and to have completed at least one ultramarathon defined as a race event longer than the standard marathon distance of 42.2 km RUNNING-RELATED INJURIES IN ULTRARUNNERS 27 (26.2 mi) within the last three years. In this study, the participants were not limited to ultrarunners who self-reported managing an injury, so as to include individuals who had experienced prevention strategies without incurring running-related injuries. Exclusion criteria consisted of the inability to speak or read English. Procedures Recruitment. Study recruitment took place over a 12-week period. An informational flyer (Appendix A) and email invitation (Appendix B) were used for the purpose of study recruitment. The primary researcher emailed race directors of ultramarathon events occurring after the targeted time period for the first wave of interview activity. This first correspondence provided race directors with a brief explanation of the study and requested their assistance in distributing the informational flyer using newsletter or social media platforms or sending out the invitation email to ultrarunners who have signed-up for ultra-distance events using email listservs. Additionally, the proposed field observation activity was described with a request to allow field observation and interview activity to take place at aid stations, medical tents, or at strategic locations at the race event. Other recruitment strategies included reaching out to publishers of newsletters distributed to the ultrarunning community, as well as social media and websites with information sharing platforms for the ultrarunning community. Additionally, contacts were gathered through word of mouth, using a snowball sampling strategy to recruit study participants. These strategies were also successful in similar studies (Holt, Homan, Youngoh, & Klein, 2014; Jaeschke et al., 2016; Simpson, Post, Young, & Jensen, 2014). The informational flyer and email invitation included the name and contact information of the primary researcher. In response to an indication of interest, the primary researcher replied to the potential study participant text, email, or telephone call in order to set up a brief phone RUNNING-RELATED INJURIES IN ULTRARUNNERS 28 conversation with the study participant. The purpose of the preliminary telephone meeting was to discuss any questions the potential study participant had regarding the study, to determine whether the individual was a candidate for the study, and to secure a date, time, and location (face-to-face or virtual) for the interview. The primary researcher contacted interested race directors in addition to an interested professional running coach to discuss logistics for distributing the informational flyer to potential study participants already signed up for specific ultramarathon events. Also, the logistics of field observation were discussed with race directors to determine where and how to interact with the study participants and to observe the ultramarathon event activities. Informed consent. All study participants were sent informed consent documents with instructions to review the documents and to contact the primary researcher with any questions regarding the information described in the documents prior to the scheduled interview. Each individual selected to participate in the interview process for the study was sent a research study enrollment email with an attached informed consent document. The informed consent document informed the study participants of their rights, the purpose of the study, and any potential risks or benefits associated with study participation. The informed consent document described the interview process and included a description of a potential second interview activity. Participants were informed that no identifying information would be attached to the data collected for the study. At the time of the interviews, the audio recording device was turned on and the primary researcher verified that the study participant had received and read the informed consent document. The primary researcher asked the participant to indicate that the information contained in the documents was understandable and that all questions had been answered by the primary researcher. RUNNING-RELATED INJURIES IN ULTRARUNNERS 29 Data collection process. Pilot testing took place as recommended by Creswell and Poth (2018), in order to ensure the methods proposed were feasible and feedback was received on the data collection instrument (i.e., interview guide). No identifying information was collected during the two pilot interviews. The pilot testing activity involved conducting both a face-to-face and virtual semi-structured, in-depth interview with two individuals who met the criteria for participation in the research study. The data collected from the pilot subjects were not included in the final participant number. Pilot field observation activity took place at the Sage Burner Trail Races in Gunnison, Colorado (https://www.madmooseevents.com/sage-burner-trail-race) and the Sawmill Trail Runs in Golden, Colorado (http://humanpotentialrunning.com/sawmill/). All data from the pilot, including the audio recorded in-depth interviews and written field note observations, were destroyed once the necessary modifications were made to the interview guide and data collection procedures. First interviews. Data were collected through semi-structured, in-depth interviews that were conducted either be face-to-face or virtually. The interviews consisted of open-ended questions, lasted approximately 45 to 60-minutes, and were audio-recorded. The interviews took place at locations conducive to a quiet environment, including public library study rooms, local university classroom space, or at the physical therapy clinic where the primary researcher is employed, allowing for the interaction to be audio recorded without interruptions. Interviews also took place using Facetime, Facebook Video Chat, Google Hangout, Skype, and Zoom for audio recording only to ensure that all data collected was non-identifying. Prior to beginning the interview, the primary researcher collected basic background information for the ultrarunner (Appendix C) including age, gender, years of running experience, frequency and severity of running-related injuries, as well as both the number and type (distance run) of ultramarathon RUNNING-RELATED INJURIES IN ULTRARUNNERS 30 events completed. The information was gathered via email as an attached and filled out by the study participant to assist a theoretical sampling strategy. Alternatively, the primary researcher completed the form via a question and answer session with the study participant at the time of the first interview. This ensured, for example, that both male and female runners and runners of differing ages were represented in the data collected. The prepared interview guide was used to help identify the strategies used by ultrarunners to prevent and manage running-related injuries. The interviews were conducted in such a way as to initially present generalized questions related to ultrarunning and allowed the participants to share openly and broadly so as to build rapport and to establish an understanding of participant stories (Charmaz, 2006). The line of inquiry then delved deeper, allowing for ideas to be expanded upon and for distinctions to be made with regard to specific prevention and management strategies utilized for running-related injuries. Two prepared interview guides were developed through the literature review process and were comprised of the following areas of inquiry: goal orientation, motivating factors, training and performance strategies, cognitive skills, pain perception, available resources, and the nature of the ultrarunning community (Appendices D and E). Based on participant responses, probing and follow-up questions were used during the interview to help interviewees reflect deeply on their experiences and to facilitate the gathering of accurate and detailed information. Charmaz (2014) described the process of intensive interviewing as a combination of focused attention and open-ended inquiry. For example, the primary researcher allowed the participants to share openly on the topic of pain and suffering before the topics of injury prevention or management of running-related injuries were approached. Later in the interview, participants were asked to provide details and to expand upon concepts that related to particular RUNNING-RELATED INJURIES IN ULTRARUNNERS 31 aspects of the emerging theory. Another interview strategy used by the primary researcher, involved prompting study participants to elaborate on recurring phrases identified across multiple interviews with different study participants. Also, the primary participant followed up on some of the shared stories by prompting the participants to elaborate on particular training or competition strategies used in ultrarunning. These examples illustrate how semi-structured, indepth interviews focused on emerging themes and supported the ideas central to the constructed theory. Second interviews. As a follow-up to the initial interview process, participants were asked about their plans for future ultramarathon event participation. Second interviews were selected based upon theoretical sampling strategies and with the intent to interview ultrarunners soon after competing in ultramarathon events to facilitate the gathering of information-rich data (Appendix E). Nine second interviews were conducted at an ultramarathon event where the study participant was competing and took place either during, or immediately after the running event. Two second interviews were conducted using the same format as the first interview and took place within a few days of participation in an ultra-distance event. Of the participants who were agreed to a second interview, four were runners who had signed up for the Stories Ultra in Colorado Springs, Colorado and five were runners who had signed up for the Behind the Rocks Ultra in Moab, Utah. One runner was interviewed both before and after running the HURT 100 in Honolulu, Hawaii. Based on the line of questioning used in the initial interviews, modifications were made to the Interview Guide for Ultramarathon Event (Appendix E), which focused more specifically on the recently experienced race-day events. In particular, pain and suffering experienced during ultrarunning, mental strategies such as self-talk, imagery, and positive thinking, as well as RUNNING-RELATED INJURIES IN ULTRARUNNERS 32 decision-making processes regarding continuing to run, scaling back, or discontinuing participation in a race event. It was determined that second interviews would be included as a part of this study to allow the study participants to share their ultra-event experiences and thought processes during or soon after event participation. The primary researcher also engaged in a dynamic and individualized process to further develop emerging themes and build additional questions into both the second interviews, as well as the interviews conducted later in the data collection process to help assess the accuracy of the proposed theory (Charmaz, 2006). As the data collection proceeded, emerging categories were taken into consideration by following up with a particular line of inquiry during the second interview or by utilizing the basic study participation information and ultrarunners with specific demographics or running experience to determine if the identified categories and themes were true (Strauss & Corbin, 1998). Field observations. With permission from the appropriate race director(s), field observation was conducted at two particular ultramarathon events where research participants were agreeable to participating in second follow-up interviews and where permission had been granted to spend time in the medical tent and aid station areas, near the start and finish areas of the race course. All data collected during field observation was non-identifying for the race participants. Field observation of ultramarathon events took place at the Stories Ultra (http://humanpotentialrunning.com/stories-ultra/) in Colorado Springs, Colorado and at the Behind the Rocks Ultra (https://www.madmooseevents.com/behind-the-rocks-home) in Moab, Utah. A field journal was used for all memo-writing activity involved with the second interview. During field observation activity, the primary researcher observed the interaction among ultrarunners and between ultrarunners and event volunteers in the medical tent and aid station areas, as well as near the start and finish areas of the race courses. The information gathered RUNNING-RELATED INJURIES IN ULTRARUNNERS 33 helped guide the focus of future interviews and theme recognition related to the ultrarunning community and ultrarunner strategies for running-related injuries. Data Management and Analysis Data analysis included managing and organizing the data, creating memos and emergent ideas, classifying codes and themes, interpreting prevention and management strategies for running-related injuries in ultrarunners, and representing the data visually (Creswell & Poth, 2018). Charmaz (2006) described grounded theory as an inductive approach that includes an iterative process between data collection and analysis. Strauss and Corbin (1998) offered a specific series of procedures to guide the grounded theory process that includes creating codes, grouping codes into concepts, and forming theoretical frameworks that include models or visual depictions of learned information. In the proposed study, the events, happenings, and instances of phenomenon for ultrarunners were given a short label referred to as a category (Strauss & Corbin, 1998) and the researcher developed, analyzed, and interrelated the categories as the theory emerged from the data (Creswell & Poth, 2018). An ontological approach involved compiling different perspectives of ultrarunners as themes developed in the research process, and therefore influenced both the point at which theoretical saturation was achieved, as well as the sample size (Saunders et al., 2017). The process of the proposed study included the development of categories and emerging theory and determined the criteria for theoretical sampling and additional data collection (Saunders et al., 2017). Data analysis began immediately after the first in-depth interview and was an iterative process. Data storage. Study participants were assigned a numeric value to ensure confidentiality and deidentification. All audio files were labeled using an assigned numeric identifier, interview location, and date the interview was conducted. The audio recordings were saved as audio files RUNNING-RELATED INJURIES IN ULTRARUNNERS 34 and the first six interviews were transcribed verbatim by the primary researcher using Dragon Naturally Speaking Version 15 and Microsoft Word. The subsequent transcribed data was reviewed by the primary researcher while listening to the audio recorded interview in a line-byline fashion. The subsequent interviews were transcribed using an automated online transcription service (temi.com). All necessary edits for both methods of transcription were performed manually to ensure the accuracy of the interview transcription. All data, including audio recordings, transcribed interviews, memos, as well as a log of data-gathering activities, were saved using a Microsoft OneDrive cloud storage and backed up using the Dedoose, Version 8.2 software program designed for qualitative research to help organize, analyze, store, and retrieve data. Inductive coding. A line-by-line coding process was performed using the software program, in which a short name was assigned to segments of data; the data was summarized, accounted for, and sorted to begin the analysis process (Charmaz, 2006). Data analysis included open coding, axial coding, and selective coding and was performed jointly with theoretical sampling as categories related to running-related injuries for ultrarunners were identified (Claire et al., 2007; Creswell & Poth, 2018). Consistent with grounded theory, the primary researcher identified the initial codes that revealed patterns and accounted for the data; codes were compared, and gaps in the data were identified, so that unexpected ideas could be further explored (Charmaz, 2006). In this iterative process, ideas emerged from the data, themes were identified, and three themes listen to your body, back on track, and mental toughness were identified with running-related injury prevention, running-related injury management, and ultrarunning participation respectively. These three over-arching themes were identified as the central features of the study (Creswell & Poth, 2018; Marshall & Rossman, 2011). RUNNING-RELATED INJURIES IN ULTRARUNNERS 35 Open coding involved the identification of significant words, phrases, and ideas from the transcription and assigned short descriptive word phrases used to develop categories of information. The software was used to organize coded information into categories, which were then analyzed using word search functions. Relationships between motivating factors and training performance strategies were identified in the literature review and solidified as patterns were uncovered. During this early phase of category development, information gathered through a literature review was utilized to develop emerging categories related to running-related injuries in ultrarunners. For example, a preliminary review of the literature helped to formulate interview questions intended to facilitate the in-depth, semi-structured interviews, categories for open coding related to the following content areas: goal orientation and motivating factors, training performance strategy, "over-reaching", nutrition/hydration/sleep habits, running form/shoe wear, cognitive strategies/coping skills, mental toughness/pain perception, coaching/medical professionals, ultrarunning community. Axial coding or focused coding was performed to interconnect the categories and involved a process of sorting, synthesizing, and analyzing the data in new ways (Charmaz, 2006; Creswell, 2014). The axial coding phase, as first described by Strauss and Corbin (1998), and later elaborated upon by Creswell and Poth (2018), explored the relationship between the data and the central phenomenon and considered each of the following factors: contributing causal conditions, responsive strategies taken, intervening conditions, and resulting consequences. Relationships between prevention strategies and management strategies were compared, as well as data supported connections between concepts that seemed unrelated, proved to be connected with regard to the aim of the study. The software was used to perform searches for terms directly related to the main themes identified for the study were performed to appreciate or refute the RUNNING-RELATED INJURIES IN ULTRARUNNERS 36 ideas shared amongst study participants. In axial coding, the over-arching themes were identified with both the theme categories and sub-themes. In later stages of data analysis and category identification, selective coding was used to verify emerging themes (Claire et al., 2007). Theoretical sampling was used to test the emergent grounded theory in a process whereby the study activity (recruitment, interview guide content, selection process for the second interviews, and focus of field observation activity) was directed by evolving theoretical constructs. (Claire et al., 2007). The software allowed the primary researcher to view the coded information and identified categories according to the participant characteristics to recognize similar information shared in response to the interview questions. Specifically, the age, gender, and running experience including the distances run during race events, as well as the history of running-related injuries experienced by the study participant was considered. For example, individuals with diverse demographic characteristics, as well as those participating in ultrarunning events with variable distances and course descriptions were intentionally recruited to test the emergent theory. Additionally, the sample size was determined based on theoretical saturation. Study participants included both male and female ultrarunners between 18 years of age and 58 years of age, representing variable degrees of experience with running and running-related injuries. A logic diagram was implemented to show the relationship between mental toughness and alterations in planned training activity versus event participation. Clustering was used to develop the diagram and to help depict overarching themes (Marshall & Rossman, 2011), and identify linkages between data groupings, as well as to recognize knowledge gaps requiring further investigation into a particular area (Marshall & Rossman, 2011). In practical terms, the process of assimilating information into a visual format played a key role in the development of RUNNING-RELATED INJURIES IN ULTRARUNNERS 37 themes essential to theory construction. Memo-writing was used as a part of data analysis, field observation, and to capture ideas related to theoretical sampling throughout the research process. Memos helped to identify patterns and themes as the data accumulated and was helpful in documenting information-rich data such as the setting observed in the start and finish areas of ultramarathon events, and to capture interactions, non-verbal behavior, and voice inflections of study participants during the in-depth semi-structured interviews (Marshall & Rossman, 2011; Stuckey, 2015). Additionally, memo-writing was used to document the thought and decisionmaking process for coding and theme development from the data (Stuckey, 2015). In order for data collection to reach saturation, data was collected until there was little new information being added with additional interviews and observation. The aim for the researcher in this grounded theory study was to fully develop or saturate the proposed model. When this had occurred, selective coding was performed: information was gathered; a figure and theoretical model were developed; and, the theory was generated (Charmaz, 2006). The function of grounded theory is to provide understanding rather than explanation, and the aim of the study is to extract meaning from the data collected and to convey ultrarunner experiences with the implementation of strategies for running-related injuries in a meaningful way that honors the study participants (Charmaz, 2006). Rigor and Trustworthiness Triangulation is a technique used to ensure that an account is rich, robust, comprehensive and well-developed; it involves using multiple data sources in an investigation to produce understanding and included both face-to-face and virtually conducted, semi-structured, in-depth interviews as well as field observations (Creswell & Poth, 2018; Henderson & Rheault, 2004). Observation took place at two different ultramarathon events where time was spent at a medical RUNNING-RELATED INJURIES IN ULTRARUNNERS 38 tent and on the race course, allowing the primary researcher to observe a large group of ultrarunners participating in the race event. In qualitative research, verification refers to the ongoing process of putting into practice the mechanisms necessary to contribute to the reliability, validity, and therefore, the rigor of a study (Janice, Michael, Maria, Karin, & Jude, 2002). A number of strategies were implemented to ensure the rigor/trustworthiness of the proposed study, including independent coding, which was performed for a portion of the transcripts to provide rigor and added trustworthiness to the study. This process is a form of triangulation by making the study more dependable, ensuring that the results will be similar if the study were to be repeated (Henderson & Rheault, 2004). Rigor in data collection. Sampling strategies ensured rigor and trustworthiness by starting with a maximalist sampling strategy that included recruitment via race directors of events differing in the distance run and type of topography. The information collected for potential study participants (Appendix C) was considered in an effort to include a representative sample of ultrarunners as described by Hoffman and Krishnan (2013). As the study developed, a theoretical sampling strategy was utilized to test emerging themes and theory development. Rigor was added by concurrently collecting and analyzing the data. Later in the study, interview questions were altered and changed based upon the findings of earlier interviews. Finally, the primary researcher aimed to establish a theory that explained the approach of ultrarunners regarding strategies for running-related injuries in a process described as, deliberation between a micro perspective of the data and a macro conceptual/theoretical understanding (Janice et al. 2002, p.13). The primary researcher attempted to identify any negative cases, which referred to data that did not fit the proposed model or theory that exists for most of the data (Charmaz, 2006; Creswell & Poth, 2018). The negative case (Creswell & Poth, 2018) and process of assessing RUNNING-RELATED INJURIES IN ULTRARUNNERS 39 theoretical adequacy (Charmaz, 2006) are accepted as a normal phenomenon and can also add trustworthiness to the proposed study. An auditing process included the creation of a tracking document established early in the research process, used to establish an audit trail describing the patterns, themes, and interpretations set forth by the primary researcher. These documents provided a rationale for the analytic steps contributing to the analysis as a whole (Silver & Lewins, 2014). The tracking documents assisted in creating an audit trail, which were used to record analytic ideas or patterns identified during the research process which added rigor to the qualitative research study (Marshall & Rossman, 2011; Stuckey, 2015). Additionally, debriefing activities were executed by individuals as a form of cross-checking the study methods of the primary researcher. This debriefing activity was conducted via discussions with individuals with experience in academics and qualitative research processes to provide oversight and feedback on data collection, interpretation, and conclusions, as well as the development of patterns, themes, and interpretations set forth by the primary researcher to add conformability to the study. Lastly, an individual with professional coaching and running experience, who was familiar with the prevention and management of running-related injuries, provided a review of the interview guides to verify that the prepared questions were relevant to the study inquiry (Appendices D and E). Rigor in data analysis. Early in the data collection process, the primary researcher, along with two individuals experienced in qualitative data analysis provided oversight and review of an established codebook. Upon completion of coding, the individual researchers contributed to the development of the codebook providing a sample of representative data. The codebook consisted of specific items that included a code name, definition, criteria for usage, RUNNING-RELATED INJURIES IN ULTRARUNNERS 40 sample text segments, and source information. From the coded data, themes were developed, revised, and refined until a consensus was reached on which codes and themes most accurately represented the essence of the data. At that point, a theme book identifying a particular number of overarching themes was used to represent the strategies presented among each of the chosen participants. Each individual researcher reviewed key excerpts of the transcriptions and agreed upon terminology to create consistency throughout the data. Once the data was revised using the agreed-upon codes and themes, a process was initiated to ensure accuracy with transcription and the themes for the developed codebook. Each researcher reported agreement with the themes and accuracy of the transcriptions. An audit trail was used to strengthen the theoretical sampling guide developed by Claire et al. (2007) for areas developed in the categories identified following initial interviews conducted. The audit trail activity described enhanced the objectivity of the study by reducing researcher bias (Henderson & Rheault, 2004). Estimating the number of participants in a study required to reach saturation depends upon a number of variables including the quality of data, the scope of the study, the nature of the topic, as well as the information obtained from participants (Aldiabat & Le Navenec, 2018; Janice et al., 2002). Consistent with the grounded theory approach, data was analyzed as it was collected until theoretical saturation was achieved, and no new themes arose from the data. Theoretical sampling was used to test the emergent categories and themes and evolving theoretical constructs for a representative sample of ultrarunners. (Claire et al., 2007). Results The sample size of 28 study participants was determined based on theoretical saturation sampling, which consisted of adjusting the inclusion criteria as the study progressed based on patterns observed in the data. Specifically, potential study candidates who indicated they had not RUNNING-RELATED INJURIES IN ULTRARUNNERS 41 experienced running-related injuries, were excluded from participation, so as to develop the back on track theme associated with management strategies and resources for running-related injuries. Also, five of the final six study participants sampled were female (Table 1) which allowed for a male to female ratio more representative of the ultrarunner population described in the literature (Hoffman, et al., 2014). As the data began to reach theoretical saturation, potential study participants were asked to complete the basic study participation information form to determine if the themes identified were consistent for different types of runners. Consistent with the grounded theory approach, the process of describing how ultrarunners approach prevention and management strategies for running-related injuries included simultaneous theoretical sampling, data collection, and data analysis, rather than utilizing a step-by-step process (Claire et al., 2007; Glaser, Strauss, & Strutzel, 1968; Saunders et al., 2017). Overview of Findings Three overarching themes emerged from the data which reflected strategies used by participants to prevent and manage injuries during their ultrarunning participation: listen to your body (Table 2), back on track (Table 3), and mental toughness (Table 4). These three theme tables describe the over-arching themes, categories, and provide sample quotations for findings from the qualitative data analysis. The first over-arching theme is listen to your body, which corresponds with running-related injury prevention strategies and resources (Table 2). The second over-arching theme is back on track and corresponds with running-related injury management strategies and resources (Table 3). The third theme is mental toughness, which corresponds with the psychological and behavioral aspects of ultrarunning participation (Table 4). A conceptual model illustrating these themes is included in Figure 1, which highlights the RUNNING-RELATED INJURIES IN ULTRARUNNERS 42 relationship between running-related injury prevention, running-related injury management, and ultrarunning participation. The theme, listen to your body describes running-related injury prevention strategies and relates to training activity, rest and recovery, event specific training, reduced impact, and event selection (Table 2). Key resources utilized for the prevention of running-related injuries include running coaches, ultrarunning community, media and publications, performance tracking tools, and exercise routines. The theme, back on track describes running-related injury management strategies and relates to pain interpretation, injury categories, time off running, fitness alternatives, and event participation decisions. Key resources utilized for the management of running-related injuries include medical professionals, sense of community, medicinal treatment, a do-it-yourself approach, and footwear. Lastly, the theme, mental toughness describes behavioral and psychological factors that influence the prevention and management of running-related injuries and relates to positive thinking, problem-solving, goal setting, personality profile, source of information, and pain and suffering. These behavioral and psychological factors influence how ultrarunners employ prevention and management strategies for running-related injuries. Stated differently, ultrarunners adopt certain behaviors and possess particular characteristics that may facilitate or hinder the implementation of strategies aimed at remaining injury-free and/or recovering from running-related injuries. Listen to Your Body" Prevention Strategies Injury prevention strategies were implemented by ultrarunners to decrease the occurrence of running-related injuries and coincided with the theme, listen to your body (Table 2), which includes both the strategies and resources associated with injury prevention efforts. Ultrarunners RUNNING-RELATED INJURIES IN ULTRARUNNERS 43 frequently used the words, listen to your body as a strategy for balancing training activity with designated recovery time. I think a good thing to ask when youre getting into this sport is, what are your goals, whether it be, I want to run a hundred miler or your goal is to win a hundred miler, or your goal is to have two great years of high performance and results or to have 20 years of running. I think its important to ask yourself that. If you want two great years of results, youre going to approach things different than if youre hoping for 20 years of doing this sport. I want to be doing this when Im 40 or 60 years old. Youre going to have more fluidity, flexibility, and listening to your body because youre in it for the long haul (Ultrarunner S1). The term is closely identified with the theme listen to your body and emerged from the data as a guiding principal for injury prevention in ultrarunning. The specific running-related injury prevention strategies employed were: training activity, rest and recovery, event specific training, impact reduction, and event selection Training considerations. Some of the training considerations discussed by study participants related to the factors of pacing, tempo, and mileage, as well as consistency with a gradual increase in training volume. The process whereby ultrarunners determined their training schedule differed considerably, and seemed to depend upon a number of factors including ability level, as well as the age and experience level of the runner. Additionally, ultrarunners described the temperatures, weather patterns, and trail conditions as influencing the training schedule. The runners in this study described a history of being involved with a number of physical activities including shorter distance running, road and mountain biking, and adventure racing, as well as a number of team sports prior to getting involved in the sport of ultrarunning. One study RUNNING-RELATED INJURIES IN ULTRARUNNERS 44 participant summarized the transition by explaining, Ive got a buddyits his quote, Youve got to be good at slow (Ultrarunner S9). The transition from a faster to slower pace was described as an adjustment for most of the interviewees. The typical weekly mileage reported by ultrarunners, one to two months before a scheduled training events was between 30 miles and 80 miles per week. Overall, runners preparing for 50K and 50-mile events described weekly mileage of 30 to 40 miles per week, while runners preparing for 100K and 100-mile events described running 60 to 80 miles per week. Despite the wide range of weekly mileage reported, ultrarunners described a typical training week, one to two months prior to a race event with many similarities. Most runners described running five to six days per week with longer trail runs of up to marathon distances on Saturdays, followed by a shorter half marathon distance on Sunday, which involved running on sore legs. Most runners scheduled at least one rest day and a day of cross-training activity and described these as part of an intentional injury prevention strategy. It would be usually about three 10 to 15 mile runs and then maybe a long 20 to 30-mile run. I know with my body and my injury experiences that I don't do as well with back to back runs. So, I do some biking in between or just total rest days, but I usually (run) a max of like five days a week and that's really pushing it for me (Ultrarunner S6). Approaches regarding the distance and intensity of workouts during the week varied considerably for runners, with differing levels of complexity and variability depending upon the approach taken for programming training activity. More than half of the runners interviewed described some form of tempo running or interval training included in their weekly or bi-weekly running routine. RUNNING-RELATED INJURIES IN ULTRARUNNERS 45 A number of runners described finding the balance between volume and intensity as a particularly challenging aspect of setting the training schedule, which was described as a source of concern with regard to the development of overuse running-related injuries. I was definitely doing more intensity, like speed work and that sort of stuff at that higher volume when I first became injured. Maybe it's just the volume and the intensity in which I was doing it. Maybe I shouldlower the volume if I'm going to be doing that, that intensity. I dont know. Its tough to know for sure (Ultrarunner S5). Ultrarunners who utilized professional coaches often described some type of periodization and reported training schedules with more interval training and varying degrees of intensity than runners who designed their own training programs. Most runners addressed the need for a training program specifically tailored to upcoming events. There was a general consensus that consistency of training was an important aspect not only for ultrarunning performance but also for the prevention of running-related injuries. A number of the interviewees also described the importance of avoiding spikes in training activity for mileage and speed work, as well as for running on hilly or mountainous terrain. A number of runners described following a 10% maximum weekly increase in mileage as a guideline. Normally I take two months off from running. And I might sprinkle some runs and just to keep fresh. And this happens every year. I literally start back over with two mile runs and maybe long runs with six miles on the weekend. Thats it. And the reason I do that is because I am base building again. So, every year for 14 years, I have gone through this process of taking some kind of break and starting back up again with base building. And again, like my miles will only increase 10% from week to week and I'll build all the way up to 80 miles a week (Ultrarunner S20). RUNNING-RELATED INJURIES IN ULTRARUNNERS 46 Rest and recovery. The over-arching theme of listen to your body was central to injury prevention strategies related to rest and recovery. Most runners described a weekly routine that included one to two days without running for rest and recovery and to off-set the repetitive nature of running activity. A few of the interviewees described negative experiences they associated with over-reaching or over-training scenarios that resulted in having to take days, weeks or months off from running. These experiences were characterized by excessive fatigue and an inability to physically recover with rest, often associated with feeling depressed or feeling an increased overall sense of stress. Runners in these scenarios were not able to link to any particular musculoskeletal running-related injury. Ultrarunner S3: In 2015 or 2016, I participated in a race series that included five races in a five-month period of time. Its basically two 50 milers, a 50 km, a 100 km, and a 100 miler, so you basically do one every month and you can say you finished the event series. And yeah, it was just too many races for me in a short period of time and I had to take some time off after that last 100-miler. Interviewer R1: How long did you take off? Do you remember? Ultrarunner S3: Not long enough to be honest. I think three months, where I just really cut back the mileage. I wouldn't call it training. I was trying to casually run a little, you know, but not training at all. Interviewer R1: So, three months. What were the indicators for you that your body needed a change in the amount of running you were doing? Ultrarunner S3: I was just very fatigued, tired, stressed, and I just wasn't getting the same joy out of running that I normally get and I just knew that at that point that it was time to step away a little bit or back off a little bit. RUNNING-RELATED INJURIES IN ULTRARUNNERS 47 Another ultrarunner described the challenges associated with balancing her training activity and event participation while being a full-time student in a doctoral program. She shared the perspective of her professional running coach on the link between physical and emotional stress. My (running) coach is big about telling me that the body knows overall life stress, not just like running related versus life related. So just recognizing that like you've gone overboard with the stress. Maybe it's like you were running high mileage and you had a project due that week and therefore you're just going to be feeling more spent by the end of the week (Ultrarunner S23). Each of the 28 runners interviewed ascribed to a tapering strategy, typically described as starting at least two weeks prior to most ultra-running events. Also, the runners interviewed took time off from running immediately after race events that varied from a one week to three week long time frame, in which they participated in no running or considerably scaled back running routines. Nearly half of the study participants described creating an off-season, lasting between one and three months in most cases and characterized by minimal running and without any scheduled race events. So, I feel like I go, you know, I kind of go pretty hard and I get pretty obsessive over maybe nine months of the year, and then just having three months off when I don't really care about performance. It feels good (Ultrarunner S11). Yeah, I would love to run every, all year round, but I don't think my body can do that anyway. I think my body needs a little time off, so it's kind of cold weather plus maybe realizing there are some real benefits to that approach (Ultrarunner S6). One runner shared her experiences with suffering two different knee injuries involving damaged cartilage, which limited her running to shorter distances. RUNNING-RELATED INJURIES IN ULTRARUNNERS 48 If I could go back and do it again, I would probably take like part of the year off every year. And I never did that (Ultrarunner S4). The ultrarunners in this study also described their pre and post-race activity in terms of rest and recovery. All of the runners described a tapering strategy that involved weaning their mileage and training activity in the weeks prior to event participation. Most runners described using the listen to your body strategy and scaling their running activity based upon whether the event was considered to be a training race or a goal race. Many of the runners described using compression socks during both training activity and event participation. The primary researcher asked one of the study participants about post-race recovery strategies and she responded, Well, I do use either compression socks or compression sleeves for my calves when I run (Ultrarunner S27). Another interviewee was asked about strategies used to facilitate post-race recovery and he described using a pneumatic compression device, I use Normatec Boots, elevate my legs, and rest. Thats really all I do (Ultrarunner S10). Event specific training. Another aspect of injury prevention in the sport of ultrarunning involves event specific training, which interviewees described as training activity aimed at preparing for the mileage, terrain, and conditions of upcoming events. While some of the challenges presented, such as temperature were less relevant to musculoskeletal injuries, training strategies for upcoming race events were mentioned by many of the subject participant. For example, one runner from Hawaii talked about her training for a hundred-mile event on a course that included rocky, root-filled terrain during a season often including afternoon rain showers. I do think having that home turf advantage was key. It helped me because I felt very comfortable in the mud. Ive run the course muddy before. You could easily slip during those conditions, but Id already done it so it wasnt too big of a deal (Ultrarunner S2). RUNNING-RELATED INJURIES IN ULTRARUNNERS 49 Ultrarunners are faced with variable terrain over long distances. As part of the field observation for this study, runners ran on snow-packed mountainous trails on steep inclines, while another event involved technical running and scrambling on sandstone with full sun exposure and sandcovered terrain. Ultrarunning is a totally different animal. You have multiple elevation gains and losses. Your footing is constantly changing on the trail. You can go from sand to rocks to roots to rock scrambling. I think gaining and losing thousands of feet of elevation fatigues your legs and can cause you to get injured because your legs get tired. Things can get twisted, things can get tight, and that adds to the difficulty (Ultrarunner S25). Reduced impact. Many ultrarunners included language describing efforts to minimize impact during running and indicated a preference for trail rather than paved surfaces. I definitely feel more beat up after I run on concrete than I do running in the mountains. It's just that on the roads, I'm doing almost the exact same foot strike every single time, you know, however many thousands of times of doing it. Whereas in the mountains, if there's a big hill, maybe I'll power hike it or if I'm going downhill, I totally change up my gait. If it's rocky, I'm doing something different. It's different engagement (Ultrarunner S11). One runner expressed his concerns about a long-term health effects of long-distance running for both the cardiac and musculoskeletal systems, I do worry about what Im doing to my heart and what am I doing to my knees? (Ultrarunner S9). There were differing opinions on the significance of running form on reducing impact during running. For runners who associated changes in running form with strategies for injury prevention, an increased running cadence with a slight change in body position characterized by a slight forward body lean were described. RUNNING-RELATED INJURIES IN ULTRARUNNERS 50 I'm on my feet all day, so I kind of consider that part training as well. You know, I'm averaging maybe 25,000 steps a day with work or family plus a little bit of training in there. And then, because I'm hands on backhoe, snow removal, welding, and logging trees I think that helps too (Ultrarunner S18). Many study participants also cited cross-training activities as a key part of injury prevention for running-related injuries, which included biking, hiking, rock-climbing, swimming, weight lifting, and yoga. A number of runners described the benefits of decreased impact, core stabilization, varying movement patterns as beneficial aspects of cross-training activity. I think cross-training is an important thing to do. Yeah, just running you know, is very repetitive, especially on the roads. On the trails you'll get some variation, but you do really need to have a strong core. People think yoga is just flexibility, which is a huge misconception. It's a lot of flexibility, which is important, especially for runners because we tend to be very in-flexible. But its also core strength. And it's not just your abs. Its also hips and lower back as well. That brings everything together and helps to stabilize everything (Ultrarunner S22). Event selection. Almost all ultrarunners described having a key race event for the calendar year. Runners described strategically setting the running event calendar based upon the amount of time between races as well as the distance of the event and specific course description. This strategy helped motivate and prepare the runners for their key race event. A number of the interviewees shared having positive first 50 km event experiences that marked the beginning of their participation in the sport of ultrarunning. It was not uncommon for runners to reflect on their first one to two years and to describe over-scheduling race events that led to the development of overuse running-related injuries. RUNNING-RELATED INJURIES IN ULTRARUNNERS 51 Yeah, so my first running-related injury was back in Colorado and I actually did get plantar fasciitis because. I ran my first 50 and then I just signed up for a whole bunch of races. I was racing almost every weekend. Every Saturday and Sunday I would be hiking fourteeners. So then by August, I just couldnt run because of the plantar fasciitis. So that was definitely an example of over doing it. I over did it (Ultrarunner S2). Many runners described scheduling shorter distance races leading up to key race events in a process that includes base-building and described setting the event calendar based upon projected fitness levels for upcoming events. One runner described capitalizing on a base-level of fitness and linking together multiple key race events, before cycling back to a five-month long period of rest and recovery. I'll do the base building, I'll do the shorter distances, I'll build up, and I'll get to that hundred-miler. And then I'll run like three or four you know, over the course of however many months. Before this break, I ran six 100-mile races, three 100 K's, and a 200-mile eventover a span of twelve months. The fitness was there, so it was just about maintaining the level of fitness that I had acquired. You can ride that out. But again, it still requires you to be smart, to listen to your body, to listen to your mind, to rest and to not always be pushing (Ultrarunner S20). Each of the runners interviewed shared stories about pushing themselves to the limit, both physically and mentally. Rest and recovery were discussed primarily as a necessary counter-part to training activity, and secondarily as a strategy for injury prevention. High points shared by interviewees involved a strategy for training activity that culminated in a finish, marking an achieved big event goal for the calendar year. Listen to Your Body - Prevention Resources RUNNING-RELATED INJURIES IN ULTRARUNNERS 52 The resources utilized by ultrarunners for injury prevention included running coaches, ultra, media and publications, and performance tracking apps. The study participants expressed differing opinions on some of the resources related to injury prevention strategies. Thoughts concerning information sharing and performance tracking devices were divided. Overall, most ultrarunners agreed that finding the right coach, trusting word of mouth and avoiding tendencies to assign self-worth based upon information shared through performance tracking tools were important strategies for injury prevention outcomes. Running coaches. Multiple runners shared high points for their running careers that involved working with a professional running coach to help achieve a goal to complete a particular race event. Several of these runners also shared low points that included one or several DNF (did not finish) experiences that they attributed in part to failing because of an inadequate training schedule, either over training or under training. Additionally, study participants described running coaches as being in a unique position to help them avoid both under-training, as well as over-training for race events. I never used a coach until about five years ago when I was training for the Leadville 100. I tried the Leadville 100 twice on my own, so then I decided I needed a coach if I wanted to finish it. So, I hired a coach and he was great. Hes actually still a friend now. I used him for about five years and he helped me finish the Leadville 100 (Ultrarunner S4). A number of runners described either having concerns related to feeling obligated to follow a particular training schedule while working with a running coach or with concerns related to overtraining as a result of feeling obligated to follow a particular program. I have found that time off and rest are critical for longevity in this sport. Ive had a number of friends who have been on really high intensity training plans that include RUNNING-RELATED INJURIES IN ULTRARUNNERS 53 coaching and who become either injured or just really burned out. I think thats pretty easy to do (Ultrarunner S1). Word of mouth. There was a wide variety of ultrarunner preferences regarding the utilization of resources related to the topic of injury prevention. The primary resource cited was word of mouth. Interviewees described utilizing friends who were runners, as well as running groups for first-hand information on strategies for staying healthy and injury free. One participant described his experience working at a running shoe store and educating runners on fit and design strategies aimed at injury prevention for running-related injuries. Interviewer R1: Can you share about your experience working at a running shoe store? Ultrarunner S21: Yeah, I just enjoy talking with people about running, so the running shoe store is just kind of a natural way to have those conversations. Interviewer R1: Do you have a particular shoe that works for you personally? Ultrarunner S21: I'm a Hoka guy for sure. The shoes I like to wear have like a four to sixmillimeter drop. So, when I get above the six, I start having problems. If I go to zero, I tend to have problems with the Achilles, so I stay away from zero drop. When I'm talking to people about it, I kind of listen to what they've been in and then try not to go too drastic on the drop. Thats the number one thing to do when youre making a change in shoe wear. Yeah, keep the changes really subtle and small. Most of the ultrarunners interviewed described the ultrarunning community as a valuable resource for injury performance strategies, however some interviewees expressed less first-hand information sharing for running-related injury prevention and management strategies. I have a few friends who run, but it's funny, I don't think we really talk specifically about how not to get injured. Maybe they don't want to tell somebody what to do or claim to RUNNING-RELATED INJURIES IN ULTRARUNNERS 54 know the best way to do it. I don't want to tell you what the right way to do it (Ultrarunner S6). Media and publications. Podcasts were frequently identified as a valuable resource for information related to running-related injuries. Specifically, podcasts with a scientific bias such as The Science of Ultra podcast (www.scienceofultra.com/podcast) as valuable to ultrarunners trouble shooting their own running-related injuries. I think probably the one podcast that has more information on running injuries than most is called The Science of Ultra (https://www.scienceofultra.com/podcasts) and they talk to medical professionals and, you know, people that have a really strong scientific understanding of the activity of ultrarunning (Ultrarunner S24). Most magazine publications, including UltraRunner Magazine (http://www.ultrarunning.com/) were described as including articles aimed toward the prevention and management of runningrelated injuries. Some runners mentioned specific event websites hosting forums with medical information related to the sport of ultrarunning as valuable resources. Reviews for ultrarunning gear including traction cleats for snowy and icy conditions were also included in discussion relevant to injury prevention for ultrarunners. Performance tracking tools. The pros and cons of performance tracking apps including apps and websites that allow runners to track the performance ultrarunners were discussed. Two of the more commonly utilized resources were Strava (https://www.strava.com) and Ultra Signup (https://ultrasignup.com). I have a watch that has its own app and I can download the information from my training. And then I also belong to Strava. I don't pay for it, but I do have the free version and that makes it accessible to my coach so she could log into Strava, and view my training RUNNING-RELATED INJURIES IN ULTRARUNNERS 55 activity. She also has a log that I send to her with my workout and she keeps track of some things there. But there's more detail in the Strava data though. It's out there if she wants to look at it (Ultrarunner S24). Some of the potential benefits of using these resources include tracking mileage and vertical gain for training purposes, providing motivation by monitoring the performance of other runners, and facilitating communication with coaches on training activity. One consistent podium finisher in the sport of ultrarunning shared her experience with managing an added sense of pressure to perform. Interviewer R1: Do you ever feel added pressure to perform using Strava? I suppose people can track your workouts, especially if youre a front runner. Ultrarunner S2: Yeah, I definitely do. I dont know how it happened, but I ended up having over a thousand followers. Let me check 1600 followers now. There definitely is that pressure sometimes but Im busy so I dont think about it too much. I just have to realize that at the end of the day, its not whats important. I was more concerned about that last year than this year. Still, there is that pressure sometimes. Exercise routines. Some runners described performing exercise routines for strengthening as a strategy for running-related injury prevention. I'm also doing some basic, I'm going to call them PT type movements. I'm doing Superman's, bird dogs, hollow cores, and various kinds of lunges, laying on my side and lifting my legs for my hips or circulating my legs from all fours, and things like that (Ultrarunner S7). With the exception of runners who routinely participated in yoga, most runners described infrequent stretching, however there were some exceptions including runners who continued to RUNNING-RELATED INJURIES IN ULTRARUNNERS 56 stretch after experiencing running-related injuries that seemed to respond to stretching. Not surprisingly, runners in the older age groups described more activity with the prevention track for running-related injuries than the younger age groups. For example, one runner in the 55 years or older category for this study, described his daily routine: I have a stretching and strengthening routine that takes over 35 minutes, but when you get to be my age, its really important to get the body moving a little bit before I go running. Then afterwards, its another five to 10 minutes for a routine that I do, which involves eccentric calf raises and quad stretches, and I've been very consistent with the rolling (Ultrarunner S24) This same runner also described an approach for his training schedule and injury management that included utilizing a running coach and physical therapist. Other study participants described using equipment, including exercise bands and foam rollers as an injury prevention strategy. I actually just ordered an elastic exercise band just to use when I am back running. Its an exercise sort of strap, to help strengthen my hips because Ive never really focused on that before. Yeah. So that's going to be a new focus (Ultrarunner S5). Back on Track - Management Strategies The back on track theme emerged from the data and relates to the goal-oriented approach taken by ultrarunners toward rehabilitation. Injury management strategies were also identified by ultrarunner study participants. These strategies were triggered by running-related injuries and were employed to promote healing and recovery of function. The focus on return to running-related activity is a key aspect of the management strategy adopted by ultrarunners dealing with running-related injuries. As depicted in the grounded theory model (Figure 1), injury prevention strategies are characterized by a departure from and return to ultrarunning RUNNING-RELATED INJURIES IN ULTRARUNNERS 57 participation. The specific running-related injury management strategies employed were as follows: pain interpretation, injury categories, time off running, fitness alternatives, and event participation decisions. Nearly all of the study participants experienced running-related injuries and described taking on specific strategies for injury management, so as to facilitate a return to running. This is an injury and I dont just want to run myself into the ground. I want to do this for life. So, I will take the time to take care of those things (Ultrarunner S16). The strategies for running-related injury management include gathering the information necessary to determine an appropriate level of running activity. Ultrarunners need to be informed in order to appreciate the complex and changing nature of running-related injury management and to determine an appropriate course of action. Pain interpretation. Most of the study participants described routinely using pain a guide for managing running-related injuries with some level of success. I realized, I'm on the healing path. It was just part of the healing process. So, then I would increase my mileage and intensity as the pain was lessening (Ultrarunner S26). Nonetheless, nearly all of the runners who had experienced time off from running described the decision-making process as difficult. This was the case for injuries incurred due to overuse and acute running-related injuries. Both scenarios were described by ultrarunners as difficult to manage. Many of the ultrarunners interviewed described experiences that involved making injuries worse, either as a result of running for too long prior to deciding to take time off running or as a result of returning to running too early after suffering an injury. One runner described his experience with a nagging hip injury that was eventually diagnosed as a stress fracture of the hip. RUNNING-RELATED INJURIES IN ULTRARUNNERS 58 I remember having some pain in the front of my hip that I was convinced was just tendinitis or some kind of soft tissue injury. I kept running and it was definitely bearable and I was able to continue running. I kept doing long runs and then it started to get worse and then it would get worse during the run, which is definitely a red flag. I know better, but if it's not getting better, I have a habit of continuing to run through something until it alters my gait. And I was probably just a month or a few weeks out from a goal race and I continued to try to run. So, to make a long story short, I ended up having a stress fracture in my femoral head. I ran basically until I was wobbling like a rodeo cowboy and that's just a pretty stupid thing to do. And then it took three months to heal. So, obviously stopping earlier would have been betterI think a lot of people who run long distances are used to suffering, used to some kind of pain, and don't want to complain about it. But you know, you don't want to be put on the shelf and not able to do what you like to do (Ultrarunner S6). Another study participant described the difficulty in determining a readiness to return to trail running and a resultant ankle re-injury. The other running injuries I've had are more like clear cut injuries where there's a definite like timeframe on when you're going to come back. Whereas with the ankle, it was very kind of hard for her to deal with in the sense that, uh, I like would feel like I'm ready. I could run on the road just fine, but as soon as I got onto something more technical or rocky, like the trails all around Colorado here, I would quickly learn that it's not ready and then, I would just feel the instability of my ankle basically. So, it was this constant, thing where I would think I was ready, when really it wasnt. And ultimately, I just wasn't giving myself the rest I needed. I didnt take the time off (running) and make the RUNNING-RELATED INJURIES IN ULTRARUNNERS 59 decision to just cross train and play it smart. It took rolling my ankle again to realize, okay, if I want this to get better, I need to actually take some time off the trails (Ultrarunner S3). Injury categories. Ultrarunners interviewed for the study described the differentiation of running-related injuries as an important injury management strategy. Most of the interviewees used specific medical terminology to describe their running-related injuries, and in many cases, discussed different injuries in categories and timeframes distinguished by the implications of the injury on their ability to run. Generally speaking, runners grouped their running-related injuries into four categories: acute injuries, overuse injuries, low back pain, and stress fractures. Acute injuries. One runner shared his experience with an acute ankle injury sustained while running an ultra-event on steep and rocky terrain. It was just always a bummer to get into peak fitness and then blow an ankle up. Now here you are two or three weeks out and fit as can be, but with an injury that really limits how much you can do. In that situation its better to be undertrained then to end up being really super fit, but unable to put proper weight the ankle. Running all super ginger will just end up causing issues somewhere else (Ultrarunner S9). Overuse injuries: Another runner was interviewed both before and after attempting a 30hour timed running-related injury. He described his thought process regarding the early and late stages of an overuse injury of the knee. This excerpt of the interview was collected four weeks prior to the race event. Things kind of hit a little speed bump on Sunday. I went out for a longer run on Monday with a half mile marathon effort and during the last couple of miles, I could actually feel my IT band flaring up. It had been fine during the whole training block. I decided not to RUNNING-RELATED INJURIES IN ULTRARUNNERS 60 bump the mileage this week even though it was going to be peak weekI've been stretching a lot. Rolling a lot. Just kind of really relaxing. Not anything (Ultrarunner S5). The following excerpt from this interview was collected two days after the race and describes the participants decision to withdraw just prior to the half-way mark of the timed event. Ultrarunner S5: I had probably 24 really good miles, like awesome. I mean I could definitely feel the IT band was going to be a mess later, but at the time I was thinking, I'm still okay. I just ran through it and then all of sudden, it was done. And it was just like in agony and pain. It was bad. Interviewer R1: I guess you live and learn on that, but going back, how might have you handled that differently? Ultrarunner S5: I mean, it's almost like I shouldn't have tried to race on it. I should have just gone into recovery mode and just scratched it and let the time pass to, to heal. But at the same time, I was in the mindset that it was going to be okay, so why not just try it? Low back pain. The definition for a running-related injury used in this study, did not include issues associated with low back pain. Nonetheless, five of the runners interviewed reported having a significant history of ongoing low back pain with periods of significant disability that affected their training activity. These individuals described reaching out to seek medical advice and treatment from medical professionals with time off running that lasted from several days to five months. Last year, I ended up tweaking my back and I spent the next six months just in excruciating pain. I went to the doctor and went to the PT and nothing really made it better until I started doing daily core work (Ultrarunner S16). RUNNING-RELATED INJURIES IN ULTRARUNNERS 61 Most runners described feeling better after running, once they had transitioned out of the acute phase of experiencing low back pain. Many of the ultrarunners in this study who had struggled with low back pain described running as an effective way to speed up healing after periodic acute episodes triggered by activities other than running. Yeah, with my back it happened right around this time last year and I was going to do my first 100K in April. So, I was really hesitant on taking time off and it was one of those, it was definitely one of those injuries where I felt better running. Um, probably because those muscles are so stiff and tight and when I ran, they would loosen up. I wasn't really...so it felt better even though it wasn't (actually better). It was one of those things. So, I kind of push through it. I took maybe a week off and then just kept going running (Ultrarunner S16). Stress fractures: A number of interviewees shared stories of stress fractures suffered as a result of their running activity, which involved the foot, the ankle, lower leg, hip, and pelvis. In each of these cases the diagnosis was made after weeks or months of training activity with worsening symptoms, eventually affecting the participants ability to run and in some cases left the individuals unable to walk without pain. The fall of 2017 I had a specific injury that ended up being a stress fracture of the talus. I had to take time off from really all types of running activity. I definitely had to alter what was going on with my training schedule to the point where that injury lingered on for at least eight weeks. I didnt feel like I was progressing and I tried to make the most out of it. At that point I had been accepted into a race called the Barkley Marathons. Its a race back in Tennessee that is very extreme terrain, and lets just say I knew having attempted the race before that this was not only going to prevent this thing from healing correctly RUNNING-RELATED INJURIES IN ULTRARUNNERS 62 but might make things a lot worse. That injury specifically, having to take the time off and its effect, psychologically on me was hard (Ultrarunner S1). Time off running. The overwhelming majority of ultrarunners with experience having to take time off for a running-related injury described the process as not only physically challenging, but also mentally and emotionally challenging. Sure, when I first injured it, it was the size of a grapefruit and I couldn't even walk on it. I was off of it for a good, like eight weeks just because I can barely walk on it. Um, and then every time I tried to like go out and do a little walking or some very light jogging, it would just swell up or be really sore and painful afterwards. So, I had to scale back my running a lot after that and then slowly I starting to work with balance on that ankle, just one-legged stuff on it and everything. So, it was definitely a challenge and it kept me cooped up for quite a while. It really sucked (Ultrarunner S16). Many of the interviewees associated time off running with feeling depressed and not their usual selves as reported by one ultrarunner, Yeah, if I can't run or if I have to hold off for a while., I get super grumpy and there is a level of dysthymia or mild depression that can happen (Ultrarunner S6). The struggles associated with fear of failure as a motivator were shared by one interviewee, Its challenging because it becomes your identity really. When youre putting your self-worth and your identity on the line it makes it super challenging to sustain an injury (Ultrarunner S9). Most runners described remaining focused on the goal of returning to running as essential to dealing with time off running. Fitness alternatives. The most common non-running activity used by ultrarunners as a fitness alternative dealing with running-related injuries was biking or riding on a bike trainer. RUNNING-RELATED INJURIES IN ULTRARUNNERS 63 Some runners also described swimming to maintain their fitness during periods of time in which they were unable to run. I just had to stop my running. There was no walking because walking would cause the pain. I could cycle. I saw a sports medicine doctor. He did some dry needling and massage type work on my hamstring. I saw him three times. Basically, I guess I had to stop running until my hamstring settled down. Well, it's really difficult because I just finished that run at the end of November and I had another 50K at the beginning of January. So, I literally could not run at all in December and even walking on the treadmill was painful, so I had to give up all running. I could cycle and that didn't bother my hamstring. Um, once I got the thumbs up for my doctor, I started alternating, walking with slow running on my treadmill. No trails, no hills, and then obviously, like loose sand or gravel or anything really aggravated it. So, I had to the majority of my training on the treadmill. Yeah, so the pain will lessen and then that's when I can increase the intensity of my workouts. So, it was a gradual reduction of pain. Once I could run on the treadmill pain free, that's when I started hitting the trails (Ultrarunner S25). Event participation decision-making. Many ultrarunners described the pros and cons associated with the growth experienced in the sport of ultrarunning. Ultrarunners who had a longer history in the sport described having a greater number of event options compared to the earlier years in the sport history. One study participant described one aspect of growth in the sport, Im encouraged by the number of people coming into the sport. That also means that the number of races that have come online has grown greatly, which is fun for me because new events are always fun (Ultrarunner S1). RUNNING-RELATED INJURIES IN ULTRARUNNERS 64 Along with the benefit of having more events to choose from, runners described a competitive process or lottery system for admission. This competitive environment for ultrarunning event sign-ups may be a significant variable in the decision-making process for whether or not to take a Did Not Start (DNS) status. I had a spot in a race called the Barkley Marathons thats very difficult to get. The popularity of some of these events and the lotteries and selection processes and everything are part of it. I had a spot in a very coveted race and I had to let go and I did. In hindsight I think it was the right decision. It allowed me to run another 100-mile race four months later in Switzerland, so I had five months to heal. That allowed me to go into a race that I had never done and that I was super excited about and I was super healthy. I had a strong race and the ankle did not bother me. - (Ultrarunner S1). Back on Track - Resources for Management Strategies The resources utilized by ultrarunners for injury management strategies were medical professionals, volunteer activity, medicinal treatment, and rehabilitation equipment. Discussions on the topic of resources utilized for injury a do-it-yourself approach, and footwear included shared experiences working with medical professionals and with mixed results using medicinal treatments. Some of the study participants described listening to podcasts with a scientific focus for ultrarunners, however one participant expressed a lack of information exchanged on the topic of injury management among ultrarunners. Lastly, footwear was described as a valuable resource for running-related injury management. Medical professionals. A number of different types of medical professionals played a role in the management strategies for ultrarunners in this study. Interviewees described seeking out acupuncturists, chiropractors, massage therapists, orthopedic physicians, physiatrists, RUNNING-RELATED INJURIES IN ULTRARUNNERS 65 personal trainers and physical therapists. One runner described choosing a medical professional based on the severity of the injury. The data collected in this study for ultrarunners seeking medical support was consistent with the information reported by Hoffman et al. (2014), that information was gathered through a survey of 1,212 active ultramarathon runners. Ultrarunner S1: I try to base who I seek out based on severity. On one hand you have the pain management doctor who Ive seen four times total. There have been multiple situations where Ive gone to see the physical therapist. I might know somethings going on but I need an expert to give me better information. Information is power right? I typically leave the office with some exercises to work on. The chiropractor Ive seen for many years uses the ART. Interviewer R1: ART is a soft tissue technique correct? Ultrarunner S1: Yes, its helpful for the psoas and different areas that seemed to be helpful for runners. The other end of the spectrum there is massage. I just got a massage this week. I dont do that as frequent as I should. Thats a maintenance thing. A number of runners struggled to incorporate the advice of medical professionals unsupportive of prevention and management strategies aligned with the themes, listen to your body and back on track. I saw a chiropractor and I remember him saying, you know, with this kind of pain and this kind of injury that you have, you should definitely not run. That's probably going to be the worst thing that you can do. Instead, I'd recommend that you bike. You should just ride your bike. I was getting this positive feedback, feeling better when was running and worse when I didnt run. And I was thinking to myself, Ive started running and when I RUNNING-RELATED INJURIES IN ULTRARUNNERS 66 run, I feel better. So, you're telling me that even though I'm feeling better, I should stop (Ultrarunner S11)? Consequently, many of the interviewees opted for a do-it-yourself approach to injury management when possible. When necessary, the ultrarunners in this study described seeking out medical professionals who focused on sports medicine with experience working with longdistance running athletes. I've found in general is it's really hard to find somebody that understands the type of training you do for ultras. The physical therapists I saw when I was still in the Navy basically just told me, you need to run less because thats why you're in pain. I don't think that's good advice for an ultrarunner because were just going to keep running. The acupuncturist that I was seeing was an ultrarunner herself, and understood the type of training that we do and I guess, understood the bad patients ultrarunners can be. So, she was more about working with you instead of against you (Ultrarunner S3). Most runners described seeking out orthopedic physicians primarily for imaging, but also for surgery, chiropractic care for spine-related issues, and physical therapy for dry needling and exercise, while acupuncture and massage therapy were used primarily for maintenance issues. Cost was a deterrent for some runners, particularly when considering medical services as part of an injury prevention strategy. Last year I got a little bit of massage. It was helpful, but I also found it to be expensive and it's kind of hard for me to justify doing it. Yeah. I think I had four massages throughout the course of the summer last year, which was an unusually big summer, culminating in that race that I described prior in the year. One over in Europe, the RUNNING-RELATED INJURIES IN ULTRARUNNERS 67 UTMB. And building up to that, I ran a difficult 50-kilometer race, and a difficult 50mile race, followed by a difficult hundred-kilometer race (Ultrarunner S11). One runner reported that she had recently scheduled an appointment with a physical therapist to receive treatment that included blood flow restriction therapy for an Achilles tendinopathy. Ultrarunner S13: I had a musculoskeletal ultrasound and I had a tear right at the insertion point of my Achilles and the heel bone. So, the Achilles tendon was fine. The doctor said, it's healthy, but the tear just looks like frayed yarn. I'm still able to run on it, but it's not comfortable and then it's afterwards, it takes a lot of time to recover. So long story short, I'm currently rehabbing that and this is my third week. I have completely backed off all kinds of running because I was still out hiking and stuff. So, I've completely backed off. I'm seeing somebody tomorrow to help with blood flow because I was told it's (a problem with) the blood flow and the insertion point there is not healing correctly. Interviewer R1: What were they doing to help with the blood flow and who will you see (for that problem)? Ultrarunner S13: I believe the person Im seeing is some kind of physician and they specialize in the blood flow restriction therapy. Sense of community. All of the ultrarunners interviewed described the ultrarunning community as a positive aspect of the sport. Many shared experiences related to volunteer activity for events, and drawing motivation from the race atmosphere. Interviewees associated a sense of gratitude, encouragement from all levels of runners, and long-term friendships, and running group traditions. In some cases, runners shared friendships that outlasted their ability to participate in ultrarunning. One ultrarunner shared about her first 50 km race experience: RUNNING-RELATED INJURIES IN ULTRARUNNERS 68 I think thats why I joined the Colorado trail-runners group and was really encouraged to do these longer distance events. Every ultrarunning race is not necessarily about winning. Its about finishing and being a part of the other people in the event. Thats another reason why it intrigued me to do the ultrarunning (Ultrarunner S2). Another ultrarunner described a long-standing tradition that involved running a 50 km event with the same group of friends each year: I ran my first ultra 1999 the Mount Damascus 50 miler and it was really a good group of people and every year we just kept coming back. The group has stayed together, we all got married, and to make a long story short this last November we ran our 20th Mountain Masochist trail run (Ultrarunner S9). Some runners expressed concerns about the rapid growth experienced in the sport of ultrarunning. A growing number of younger runners and an expanding number of ultra-events to choose from were cited as positives for ultrarunning, while media hype, a focus on performance over participation, and added pressure from media hype and performance tracking tools were often described as negatives for the sport. One study participant stated, Im encouraged by the number of people coming into the sport. That also means that the number of races that have come online has grown greatly, which is fun for me because new events are always fun (Ultrarunner S1). A number of the volunteers at the ultrarunning events where the primary researcher participated in field observation described volunteer activity as a way to connect with the ultrarunning community during time off running, to support the other runners, and to earn points for future race events. RUNNING-RELATED INJURIES IN ULTRARUNNERS 69 I love volunteering and just being a part of the race environment and you learned a bunch. So even before I ran my first hundred miler, I volunteer at two or three of them and did pacing or worked an aid station. It's good to get into the environment, not just to see what other people do, but to just kind of get inspired. Seeing people push through these events can be good if your motivation is a little down; it can really spark your interest again. Like, all right, I can get back out there now. (Ultrarunner S3). Medicinal treatment. Both field observation and interview activity supported the use of medicinal treatment by ultrarunners including non-steroidal anti-inflammatory (NSAIDs) medication, cannabidiol (CBD) oil, as well as epidural steroid injections (ESI). During my last race at like mile 55, I tripped on something. Somehow my leg got caught behind me and I strained that hip flexor. I'm like, oh my God, that hurts and Im going to be reduced to walking the last like 55 miles of this race. I mean, it sounded terrible. I had painkillers with me, but I relied on Tylenol and CBD and it dulled it enough that I could run the downhills and run some of the flats. But that kept me going. And then the last 20 miles I was out of everything and I just had to gut it out (Ultrarunner S10). Nonetheless, some runners did seek out medical intervention during particularly debilitating episodes of acute low back pain. Ive gotten to the point with lower back pain where I was just unable to walk and reduced to basically rolling through the house. In those instances, the pain management injections were quite helpful. The process involved injections of high-powered anti-inflammatories with fluoroscopy for the back. In the scenario that can take months. Were talking about gradual progression back to activity (Ultrarunner S1). RUNNING-RELATED INJURIES IN ULTRARUNNERS 70 Footwear. Many of the ultrarunners in this study described shoe wear and orthotic inserts, as well as taping and compression socks as a key resource for managing running-related injuries. Several of the interviewees shared stories of going to great lengths to purchase as many pairs of a particular type of running shoe that had been discontinued. One runner described a doit-yourself approach to manage a running-related ankle injury that included balance, strength training exercises, and adherence to a proven shoe wear design. Typically, the ankle sprains I had would be a real quick rollover. I never had to have medical attention on it, but my understanding is it would have been considered a 2nd degree sprain because I would get the bruising and thats when I knew I really turned it over. So, then I would just spend tons of time, over a couple of weeks using ice, stretching, and then using a wobble board, and doing strengthening exercises on it. And then I found a particular shoe called the Montrail Hardrock. Once I found that shoe, I went four or five years because I just did not turn my ankle in that particular shoe. Then they quit making it, so I had to go online and buy every pair I could get ahold of. (Ultrarunner S9). One runner mentioned having benefitted from orthotic inserts that helped her return to running after a particular running-related injury, The orthotics were huge for meI had plantar fasciitis or something along those lines and the orthotics were a huge help (Ultrarunner S4). Some of the ultrarunners observed at race events used tape on their feet to help prevent blisters or for arch support and many of the runners also wore compression socks or compression sleeves on their lower legs during race event participation. Mental Toughness Psychological Factors RUNNING-RELATED INJURIES IN ULTRARUNNERS 71 The majority of study participants interviewed identified the mental challenges associated with ultrarunning participation as the crux of the sport. The mental toughness theme emerged from the data and includes both psychological and behavioral factors. The psychological factors identified were personality profile, source of motivation, and pain and suffering, while the behavioral factors identified were positive thinking, problem-solving, and goal setting. Study participants discussed a number of psychological factors related to mental toughness including their self-described personality profiles, source of motivation, and ideas on pain and suffering. These factors influenced ultrarunner prevention and management strategies for running-related injuries and served as either facilitators or barriers depending upon a number of variables including the study participants experience with ultrarunning and running-related injuries. Personality profile. Many of the study participants referred to themselves as having a type A personality. Ultrarunners also used the terms high achieving, task-oriented, and driven to describe the individuals in the sport. One interviewee shared her struggle with time off running and indicated the following: The time off is really hard because I didn't have a very healthy relationship with running at that time. I was having a really tough year that year and felt like, I have to run. I have to train. I have to keep on task. I can't miss like 10 miles. And, so it was really hard for me to miss and feel like, oh my God, I'm setting myself back by like months. And, of course I wasn't but when you're in it, it's magnified. So yeah, I think were to encounter that same thing again, I would handle it better. It wouldn't hit me nearly as hard as it did. Keeping a big picture perspective that season was a tough one (Ultrarunner S8). RUNNING-RELATED INJURIES IN ULTRARUNNERS 72 Similarly, a number of runners identified event over-scheduling and failing to listen to their bodies during training activity as the cause of sustained overuse injuries early in their ultrarunning carrier. One ultrarunner expressed the following: I think I tried to do too many races per year early on, training too hard and not taking enough breaks. That probably wasnt great for my knees Im sure. If you want to do it for the long term, its better to slow down in that regard (Ultrarunner S4). You don't worry about the time. I think a lot of, I think I see a lot of runners that are used to competing in shorter distance races are so focused on their pace and time to finish that they lose sight of the fact that it's just about finishing the distance not where you finished in the pack (Ultrarunner S7). Source of motivation. Ultrarunners described a number of motivating factors related to the theme mental toughness. Some of these involved a sense of curiosity related to self-discovery and finding out what is physically and mentally possible, a desire to engage in adverse scenarios, and a desire to experience challenges that involved pain and suffering, as well as a willingness to face a fear of failure. The struggles associated with fear of failure as a motivator were shared by one interviewee, Its challenging because it becomes your identity really. When youre putting your self-worth and your identity on the line it makes it super challenging to sustain an injury (Ultrarunner S9). One study participant included drawing from external motivators such as listening to music while running, and focusing on the opportunity to earn belt buckles from different race events as a part of mental toughness: If I'm just in a rut of some kind and need to get out of my own head, I'll just jam on the music for half hour and that usually helps. If I think I'm getting a little too negative. Ill RUNNING-RELATED INJURIES IN ULTRARUNNERS 73 start thinking about the finish line and you know, getting a buckle, even if it's like an extrinsic thing to get you through, something tangible that can help. I'm okay with the extrinsic thing to help get you through a tough spot (Ultrarunner S21). Study participants also described adopting a me against me strategy for ultrarunning events and included specific examples pertinent to management and prevention strategies for running-related injuries. One runner recalled thinking, You're not really racing anybody but yourself (Ultrarunner S11). Another study participant shared her experiences with reconciling a desire to compete with a me against me mindset: I really try to hold the reins back for the first half. Once I hit that halfway point, I'll pick people off who are ahead of me. I just focus on passing that next person ahead of me. I'm not being cocky, I promise, so it becomes about knowing theyre further ahead, they are running faster than me and I have to work towards the end. It takes my mind off of thinking, oh my gosh, I've only gone 19 miles. I have so many more to go. I really enjoy pushing myself and seeing what I can do. Im the type person that will make that goal and work towards it. Also, I'm really a social person so its also about the people and letting them challenge you (Ultrarunner S23) Pain and suffering. The ultrarunners in this study were able to articulate the difference between pain and suffering associated with normal running activity from injurious pain, associated with a running-related injury. Most of the ultrarunners interviewed, expressed being either reluctant or altogether unwilling to discontinue race event participation due to pain considered to be a normal part of participation in the sport. Many interviewees described the pain and suffering experienced in ultrarunning as an appealing aspect of the sport. RUNNING-RELATED INJURIES IN ULTRARUNNERS 74 I cant describe it, but there are moments that feel like I'm not going to be able to get through them, but I do. And it feels really good to have gotten through those moments to a better feeling. You know, there's a, there's a saying that the group I run with has, it's like if you're going through hell, keep going because there is something on the other side of it. And what's on the other side of it feels really satisfying and that feeling is magical (Ultrarunner S10). Some of the study participants did describe experiences that involved discontinuing running in an ultramarathon event due to injurious pain. Others conceded that there were circumstances involving injurious pain that would cause them to drop out of ultramarathon events. Perhaps more relevant to this study, was that nearly all study participants described having difficulties determining at what point to alter their running activity due to injurious or potentially injurious pain. One runner described his decision to continue running the last 10 miles of a 100-mile event: It was when I was at mile 90 and, and you know 90 is so close to a hundred, but you still have to run 10 miles, which is not nothing, you know. And I was in severe pain with that hip flexorand I came up, this rise where I could walk. I had been walking, I got to the top and I saw this long descent stretching out ahead of me and I knew how bad it was going to hurt because it was like a stabbing pain in the muscle, not in the joint. I knew it wasnt the joint. It was just like a tendon muscle thing. So, I had 10 miles to go. Maybe this is an injury and not just some pain thing, but I've got a long time to recover so screw it. I'm just going to gut it out (Ultrarunner S10). Many of the ultrarunners interviewed in this study described the significance of selfdiscovery and finding out what is physically and mentally possible, a desire to engage in adverse RUNNING-RELATED INJURIES IN ULTRARUNNERS 75 scenarios and experience challenges that involved pain and suffering, as well as a willingness to face a fear of failure. One ultrarunner described his thoughts about curiosity and a desire to explore his potential as a runner: Pain and suffering are actually kind of why I participate (in ultrarunning) in the first place. Masochism. Just pushing it to see where the limits are going to be. I think the motivator is curiosity. I want to see what I'm capable of doing. What is 80 miles? How am I going to perform? What's it going to feel like (Ultrarunner S18)? Another study participant discussed the parallels between adversity faced: It's like learning something about the nature of life. And a lot of ultrarunners have said, you know, a hundred mile race is like life in a day, going through all of it, the ups, the downs, the challenges, getting through it, you know, you come into contact with the essential parts of yourself because they really get tested (Ultrarunner S10). Mental Toughness - Behavioral Factors In addition to psychological factors, study participants described either adopting or having a propensity for a number of behavioral factors related to the mental toughness theme. These behavioral factors included positive thinking, problem-solving, and goal setting. These factors influenced ultrarunner prevention and management strategies by either serving as a barrier or facilitator to the prevention and management strategies utilized for running-related injuries. Many of the study participants reported that experience in the sport of ultrarunning as a critical component of using mental toughness to help rather than hinder the prevention and management of running-related injuries. Positive thinking. A number of mental strategies focused on positive thinking were cited as helpful in ultramarathon event performance: positive self-talk, mantras, imagery, and a me RUNNING-RELATED INJURIES IN ULTRARUNNERS 76 against me approach, as well as both associative and dissociative strategies. A number of runners shared examples of positive self-talk used during race events. One interviewee reported, The positive thinking is like, you can do this. You're amazing, you're strong, you've put in all the training, just keep going (Ultrarunner S7). Another runner described his mindset while ultrarunning in race events, I think so much of ultrarunning is mental. I really do. Two decades ago I dont know if I would have said that. Anyway, you train your body, youve got to train your mind to be uncomfortable. You know that might include cold, wet, hungry, whatever. Youve got to train your mind to know this is doable (Ultrarunner S1). A different runner shared two dissociative strategies she used during ultradistance running: Ultrarunner S23: I have a few mental tricks that I do, like I never fully think about the mileage until I'm halfway. So, obviously you know where you are because you go through age stations, but I just don't really think about it during the first half of the race, you know? Interviewer R1: Right. So, you get to mile 25 of a 50 miler before you tell yourself okay, here's where I'm at mileage wise? Ultrarunner S23: Right. Here's where I am. Now I just have to run it back and Ill be okay. Ive done 25 miles many times. It sounds kind of silly, but instead of focusing on what is tired, Ill think about what feels good right now. So, I think my quads are actually feeling pretty good. But Ive found that the game is harder to play towards the end of a race, because you're limited to say, forearms feel good. It's not fun anymore. RUNNING-RELATED INJURIES IN ULTRARUNNERS 77 Interviewees seemed to most naturally connect positive self-talk strategies with management strategies for running-related injuries. For example, one runner summarized her experience with an ankle injury: The key is to not give up. Every ultrarunner Ive talked to had dealt with injuries. Um, but the main thing is, are you going to keep lacing up right? Just be positive. You know, injuries, we can all work through them. An injury doesnt necessarily mean the end of a running career. You just have to be patient and you just have to take time off, and do your research. You know, it's, it's not hard to strengthen stability muscles. It takes like 15 minutes. I do it in my jeans while I'm watching television. It just takes a couple of minutes. Are you going to keep waking up and hitting the trails or are you going to let that injury define you and give up amazing experiences? - (Ultrarunner S25 One runner described the decision-making process regarding whether or not to continue running in an ultramarathon event at an aid station, which included attempts to use positive self-talk and talking with an aid station worker about his knee pain: Ultrarunner S5: I was going out for my first purple loop. I felt pretty good. I kind of complained at the aid station a little bit too much. I was trying to tell myself to just forget about it, to just go out. It's manageable. The pain was still manageable. Just manage it. Interviewer R1: And did you feel like that sort of grounded you in the problem somehow? Ultrarunner S5: Oh totally. Yeah. I thought way too much about it. Yeah. Problem-solving. Ultrarunners in this study described using problem-solving strategies not only for addressing challenges experienced during training, nutrition, hydration, and temperature regulation but also for prevention and management strategies related to running- RUNNING-RELATED INJURIES IN ULTRARUNNERS 78 related injuries. One study participant living in Nevada described the process of building tolerance to running in the extreme heat of the desert: Yeah, I'll tell you for being out here in Las Vegas, you know, one of the keys to our training is actually the heat acclimation training. Once the summer comes, you have to actually build up. It's almost like when you begin a training cycle and have to build your training, you have to start very carefully. It feels slower and you have to purposefully, you have to work runs into the middle of the day and in the heightened times in order to get through it. I mean we ran when it was 114 degrees this year (Ultrarunner S15). Another ultrarunner summarized his experience with mental toughness as it relates to problemsolving, and highlighted the importance of preparation, planning, and adaptability for success in order to participate in the sport of ultrarunning: And it stops becoming about running and it just starts becoming, you know, just stubbornness and mental fortitude. And I think that's what I find fascinating is that, you know, every race is so different. I feel like I've learned a lot from each one that I go into and each one that I do. Its such a wild learning experience. Yeah. You, you, you have an idea of what you're getting into and then maybe it, maybe I could line up at a 10K race and Im more or less guaranteed to finish it. If line up at a marathon, it might take me awhile. I'm going to finish it. When you line up at a hundred miler, there's a better chance that you're not going to finish it. There's a lot of variables from weather that involves heat or freezing cold temperatures or a torrential downpour for hours and hours or dealing with a hamstring strain or whatever. There are a multitude of things that you can get into out there (Ultrarunner S11). RUNNING-RELATED INJURIES IN ULTRARUNNERS 79 One study participant described how his experience with a nagging overuse running-related injury that typically became aggravated during downhill running, It seems like all my big events were plagued by something and I bet thats pretty common with ultrarunning (Ultrarunner S9). This runner was successful in finishing a 100-mile plus distance ultramarathon race estimated to have over 55,000 feet of vertical gain, and shared a training strategy to minimize downhill running. The study participant incorporated a cross-training activity designed for maximum vertical gain with minimal downhill loading forces: incline treadmill running at 15% grade with 3,200 feet of vertical gain earned in just an hour-long period of time; and snow-skiing that involved uphill skiing with snow skis set up to allow for climbing steep terrain, with a morning routine that included two trips up and down the mountain, each involving 2,500 to 3,000 feet of vertical gain. Study participants described frequently using problem-solving skills, not only for ultrarunning participation, but also for both running-related injury prevention and management strategies. Significantly, nearly all of the runners interviewed described problem-solving as an integral part of both the listen to your body prevention strategy, as well as the back on track management strategy during ultrarunning training activity, however study participants admitted to abandoning different aspects of both strategies during ultramarathon event participation in order to finish and improve their performance during race events. Some of the interviewees described using B events for training and to prepare for A events and described differences in their mindset with regard to prevention and management strategies for running-related injuries. One runner who finished first in her age group, described her experience during an event where the primary researcher conducted field observation: RUNNING-RELATED INJURIES IN ULTRARUNNERS 80 I wasn't sure if I was going to make it to the finish today. You know, my glutes and my IT band were hurting and you know, you just have to grit it out. I mean at that point, if Im racing, I'll go until I can't go, or my knee gives out (Ultrarunner #28). Specifically, these study participants shared stories in which they abandoned different aspects of the listen to your body prevention strategies including a felt need to rest, as well as the back on track injury management strategy that involved focusing on the problem area in order to determine the nature of the injury and appropriate level of running activity in order to finish ultramarathon events. Another ultrarunner contrasted his mindset regarding injury prevention and management strategies during training activity with event participation: Interviewer R1: So, during training there is a process for dealing with potential injuries, but during an event, theres a different sort of decision-making. Can you explain the difference? Ultrarunner S9: Yeah, so if youre just training you can ease up, but if youre in the event youre in a fight now. You have two choices: you can quit, whichthats hard to do; or you can finish. I try not to scar my mentals I call it. Interviewer R1: Whats the scar? Ultrarunner S9: Yeahif you set your mind to do something and then you dont do it, the door the next time opens and its a little easier to quit. Goal setting. Some of the interviewees who shared event participation stories described the benefits of focusing on short term goals in order to achieve long term goals. For some runners, this included focusing on the present tense in order to complete longer than marathon distance events. One runner described his interpretation of mental toughness, which involved focusing on one step at a time during training and event participation. RUNNING-RELATED INJURIES IN ULTRARUNNERS 81 So, you're not thinking about a hundred miles, Im thinking about one mile and then another one mile. That's a lot easier to manage. I'm just going to break the race up into segments by focusing on aid station to aid station. I'm going to meet my crew at the next aid station, So, you want to just breaking the big race up in the smaller pieces (Ultrarunner S14). One study participant described setting a goal to focus on the present tense during long runs: I think youre just focusing on being in the moment and youre not trying to put all of those interpretations ofIm not good enough, this or that hurts, I didnt train enough, that kind of stuff in your mind. Youre thinking more aboutpay attention to your breath, or how your foot falls, that kind of thing (Ultrarunner S4). This included establishing shorter training events in order to prepare for goal races for the year. In some situations, the runners described focusing on finishing race events and were unable to effectively evaluate the long-term impact of pushing through pain. Discussion The purpose of this study was to develop a grounded theory to describe how ultrarunners approach prevention and management strategies for running-related injuries. Data collected through interviews conducted with 28 ultrarunners and field observations from two ultrarunner competitions were gathered and analyzed to provide a deeper understanding of the physical and mental strategies implemented by ultrarunners for running-related injuries. Additionally, the study provided insights into what resources ultrarunners utilize in the prevention and management of running-related injuries. This research has wide ranging implications, but focuses on the following study findings: relevance of listen to your body to prevent overtraining and to reduce training loads, an injury-specific back on track approach focused on RUNNING-RELATED INJURIES IN ULTRARUNNERS 82 identifying the nature of the injury in order to facilitate a return to running activity, and the significance of mental toughness on running-related injury prevention and management strategies including behavioral and psychological factors utilized for participation in the sport of ultrarunning. Listen to Your Body The ultrarunners interviewed in this study emphasized the importance of, listen to your body in order to apply injury prevention strategies to training activity. In particular, more experienced and higher level ultrarunners described adhering to a train by feel approach, and many runners avoided setting a particular number of miles as part of their weekly running schedule. Instead, these ultrarunners focused on perceived level of intensity, and training that focused on elevation, terrain, and vertical gain as well as time on feet. This strategy allowed runners to meet their ultrarunning performance goals while incorporating injury prevention strategies that balanced periods of high intensity training with necessary rest and recovery, while accounting for event specific training. Ultrarunner preference for a training approach that includes the concept of listen to your body is supported by the literature which describes the need to allow for a cycle of training load, followed by adequate rest and recovery (Kellmann et al., 2018; Kreher & Schwartz, 2012). These authors contrast training activity that balances running activity with adequate amounts of rest and recovery from excessive training loads plus inadequate recovery, which can lead to risks associated with adrenal fatigue, and over-training syndrome. Both adrenal fatigue and over-training syndrome are described as having adverse effects on multiple body systems (neurologic, endocrinologic, immunologic), along with uncharacteristic mood changes (Kellmann et al., 2018; Kreher & Schwartz, 2012). RUNNING-RELATED INJURIES IN ULTRARUNNERS 83 Study participants described slower running speeds, in addition to limiting weekly mileage as ways to mitigate overuse injuries, which is consistent with the findings of other research studies (Hoffman & Krishnan, 2013; Knechtle, 2012). The listen to your body approach for running-related injury prevention also included efforts to reduce overall impact by running on trails rather than concrete or asphalt, and included cross-training for diversity in movement patterns. The primary injury prevention strategy employed by ultrarunners involved making modifications to training intensity, and use of alternate training activities, which aligns with weekly training distance being the primary modifiable risk factor for running-related injuries (van der Worp et al., 2015; Van Gent et al., 2007). Interestingly, Hoffman et al. (2014) collected survey information on 1,212 ultrarunners and discovered that training distances reported for those who had and had not sustained an injury were similar. This suggests that there may be determinants of injury other than distance run. In the same study, those who identified themselves as having experienced a running-related injury were younger, less experienced runners who had spent a greater proportion of their exercise time at high intensity (Hoffman et al. 2014). More experienced ultrarunners described the utilization of running coaches as a useful injury prevention strategy. Online resources, such as Ultra Signup allowed ultrarunners to implement a strategic approach toward event planning that included using B events for training and to prepare for A events that were aligned with ultrarunning or other outdoor adventure goals for the year. Other resources identified as useful by ultrarunners to augment running-related injury prevention strategies included information gathered by word of mouth within the ultrarunning community via running groups, friends who run, and race reports, as well as podcasts with shared stories. Back on Track RUNNING-RELATED INJURIES IN ULTRARUNNERS 84 The grounded theory presented connects the injury management strategy for runningrelated injuries with the term, back on track and a process focused on the goal of return to running. The injury management strategies employed by ultrarunners involved problem-solving to identify the nature of the running-related injury, and to determine whether or not to continue running activity, in addition to establishing a way to maintain some level of fitness. The types of running-related injuries experienced by ultrarunners in this study were consistent with what has been reported in the literature and included knee injuries (including iliotibial band issues), back injuries, calf strains, hamstring strains, Achilles tendinopathies, and ankle sprains, as well as plantar fasciitis (Hoffman & Fogard, 2011). Hoffman and Fogard (2011) reported that 7.5% of finishers and 15.8% of non-finishers who participated in an ultramarathon event reported having an ongoing running-related injury prior to event participation. Therefore, the process whereby ultrarunners seek to differentiate the transient aches and pains from running-related injuries potentially worsened by continued running activity is a critical component of both injury prevention and management strategies for running-related injuries. Study participants reported using pain interpretation as a guide for managing running-related injuries during training activity, but admitted to abandoning the strategy during ultramarathon event participation in order to achieve the goal of finishing the race. The participants in this study reported having a lower incidence of running-related injuries compared to other study findings for ultrarunners. Each of the ultrarunners who participated in this study completed a questionnaire which asked them to select a statement that best described their experience with ultrarunning and running-related injuries. Of the 28 runners who participated, four indicated that they had not experienced any running-related injuries since the time they began ultrarunning, while 18 chose the response, I have experienced a running RUNNING-RELATED INJURIES IN ULTRARUNNERS 85 related injury less than half of the years Ive participated in ultrarunning, A mere six of the 28 (21%) of the study participants reported having experienced at least one running-related injury most (50% or more) of their years participating in the sport. In a longitudinal study involving 1,212 active ultramarathon runners, the majority of participants (65%) had a running-related injury resulting in a significant loss in training days (median of 14 days) within the previous year (Hoffman, Krishnan, & Lucia, 2014). The lower incidence of injuries reported by the ultrarunners in this study could be related to differences in how running-related injuries are defined in the literature or poor recall regarding past experiences with running-related injuries. Ultrarunners experiencing a running-related injury, must also determine whether or not it is necessary to alter their race event schedule. The ultrarunners interviewed for this study were accustomed to the process of preparing for upcoming race events. Perseverance, and planning culminate in an event experience that also involves on the fly learning and the ability to adjust to unforeseen circumstances. The medical professionals working with ultrarunners should emphasize the educational component and communicate an injury specific and goal-oriented plan that focuses on a return to running. Covassin et al. (2015) identified that an athletes motivation to participate in the management of a sports-related injury in the rehabilitation setting as a critical factor in returning to preinjury levels. It may be appropriate for the plan of care to include a wait and see approach regarding the timeline for return to running, presented with the understanding that on the fly learning can facilitate a swift and full recovery. Another side of the sport discussed by study participants related to mental health and struggles with depression. Buck, Spittler, Reed, and Khodaee (2018) reported a 20% positive screen for depressive symptoms for participants of an ultrarunning race event. In this study, some of the study participants described the benefits of distance running as helpful to alleviate feeling RUNNING-RELATED INJURIES IN ULTRARUNNERS 86 depressed or identified time off running with undesirable mood changes. Therefore, it is critical for coaching and medical professionals to consider appropriate outdoor cross-training activities such as walking, hiking or cross-country skiing and to consider consulting counselors or primary care physicians for guidance regarding mental health challenges presented during necessary time off running. Additionally, many of the participants in this study routinely connected with other ultrarunners through running groups, and running with friends. Most of the participants in this study associated time off running for running-related injuries with feeling disconnected and therefore, it could be useful for these runners to listen to podcasts that highlight ultrarunner experiences of comeback stories for common running-related injury. Rochat,Hauw, Gr, and Seifert (2018) performed an inductive content analysis and concluded that successions of questions and solutions are meaningful not only for building, enriching, and adjusting their trail running activities, but also for coping with constraints related to running injuries. In this study the authors demonstrated information sharing via a particular online trail running focused platform as an effective way to offer suggestions for alterations in training schedule (Kimmerle, Gerbing, Thiel, & Cress, 2012; Rochat et al., 2018). Also, individuals recovering from running-related injuries who are accustomed to running groups, may benefit from event volunteer activity as a way to stay connected to the ultrarunning community. Mental Toughness The mental toughness theme emerged from the data collected through the interview and field observation activities with specific behavioral and psychological factors that include characteristics inherent to ultrarunners, as well as skills developed through participation in the sport of ultrarunning. Jaeschke et al. (2016) provided an expanded understanding of mental toughness that described the challenges associated with ultrarunning and defined mental RUNNING-RELATED INJURIES IN ULTRARUNNERS 87 toughness as, the ability to persist and utilize mental skills to overcome perceived physical, psychological, emotional, and environmental obstacles in the relentless pursuit of a goal (Jaeschke et al., 2016, p. 64). Significant to this study, is the influence of these performance factors on both prevention and management strategies for running-related injuries in ultrarunners. Specifically, the ultrarunners in this study described rallying their efforts to complete race events by applying positive thinking, problem-solving, and goal setting strategies. These endurance athletes also described drawing from key aspects of their personality and different sources of motivation, in addition to embracing ideas about pain and suffering in order to finish the race events. All of the study participants shared stories in which they abandoned different aspects of the listen to your body prevention strategies including a need to rest. Study participants also described abandoning the back on track injury management strategy in order to finish ultramarathon events. Psychological factors. The ultrarunners in this study described themselves as being type A, over-achieving, stubborn, and addicted to exercise. These findings were consistent with those of Lastly, Buck, Spittler, Reed, and Khodaee (2018) who used the Exercise Addiction Inventory to study the psychological attributes of ultrarunners and reported that 20% of respondents screened positive for exercise addiction concerns. This self-identified personality profile may serve as a barrier to strategies related to both the listen to your body and back on track themes. It is reasonable therefore, that ultrarunners who are addicted to exercise may have difficulty implementing prevention and management strategies that require time off running for rest and recovery or tissue healing. Therefore, it may be beneficial for running coaches and medical professionals to educate ultrarunners on the risks associated with over-training, overuse injuries, and timelines necessary for healing. RUNNING-RELATED INJURIES IN ULTRARUNNERS 88 In this study, ultra-runners identified a desire to achieve personal goals, along with a curiosity to explore their physical and mental limitations as a primary source of motivation, which was also reported by Roebuck et al. (2018). Specifically, the ultrarunners in this study expressed a desire to experience pain and suffering, to face their fears related to failure to accomplish difficult tasks, and to explore the limits of their human potential. This was particularly the case with more experienced runners, including runners who described their motivation to participate in 100-mile events. The me against me strategy identified through the interviews conducted is also supported by literature that describes ultrarunners as goal-directed, but not necessarily win orientated (Ogles & Masters, 2003). Some of the interviewees described the me against me approach as helpful in avoiding perceived pressures associated with social media and performance tracking applications. Additionally, interviewees described ultramarathon event learning experiences as helpful for the challenges presented in family and work life. Study participants discussed these internal motivating factors and described a desire to experience self-discovery, adversity training, and the learning experiences presented through adverse circumstances presented in ultrarunning participation. The interviewees were able to describe the difference between pain and suffering associated with normal training from injurious pain, described as more isolated to a particular body part and associated with a runningrelated injury. In this study, more experienced ultrarunners acknowledged the significance of pain interpretation for decision-making while dealing with running-related injuries and embraced pain and suffering as necessary part of finishing ultrarunning race events. Nonetheless, most of the ultrarunners interviewed did describe having difficulties determining an appropriate strategy for running activity while dealing with a potential running-related injury. RUNNING-RELATED INJURIES IN ULTRARUNNERS 89 Behavioral factors. Most of the runners interviewed acknowledged the need to stay positive during periods of time off running resulting from running-related injuries. A few runners acknowledged the significance of combatting negative thoughts experienced during time off running. Key strategies shared by the ultrarunners in this study included positive thinking and a me against me approach. The ultrarunners in this study described using a number of mental strategies including self-talk and mental imagery consistent with the findings of Holt et al. (2014), Jaeschke et al. (2016), Multhaupt & Beuth (2018), and Simpson et al., (2014). Runners also reported using associative strategies such as body scanning, as well as dissociative strategies such as listening to music. In this study, there was no preference for associative versus dissociative strategies among male versus female ultrarunners as reported by Philippe and Seiler (2005). Multiple authors have identified running distance as a risk factor for injury (van der Worp et al., 2015; van Gent et al., 2007), yet Hoffman (2014) reported that the injury rate was not higher among ultramarathon runners competing at the 161km distance compared with ultrarunners competing in shorter distance ultrarunning events. Study participants consistently described problem-solving as an essential strategy for overcoming the mental and physical challenges associated with ultrarunning. Future research should investigate the effectiveness of problem-solving strategies employed by ultrarunners to offset a high volume of training. A number of study participants associated positive thinking with favorable outcomes and negative thinking with unfavorable outcomes for ultrarunning activity. These shared stories were consistent with the findings of Alschuler et al. (2019) who reported that the use of adaptive coping strategies over maladaptive strategies proved to be the most effective way to cope with pain. Significantly, a number of the higher performing interviewees described combining both RUNNING-RELATED INJURIES IN ULTRARUNNERS 90 associative and dissociative strategies with positive thinking to focus on body parts that were feeling good and ignoring areas of the body that were hurting. Interestingly, each of the four interviewees who described using this strategy had experiences as podium finishers for ultrarunning events. Future research should investigate the connections between psychological strategies, pain perception, and strategies to help ultrarunners differentiate transient aches and pains considered to be a normal part of the sport from significant running-related injuries potentially worsened by running activity. Study Limitations The study sample of 28 included individuals living in eight different states; however, 20 of those individuals were from the state of Colorado. Therefore, the results may not be applicable to all ultrarunners. Use of a larger national and international sample may have resulted in alternate themes. The primary researcher is a physical therapist with experience treating ultrarunners and with opinions on treatment strategies for running-related injuries, thus biasing the study results. However, strategies were included in the study design to mitigate bias during data collection and analysis including the use of a field journal, auditing process, memo-writing, and triangulation. Due to the multifactorial nature of running-related injuries, unexplored possibilities for further examination across the spectrum of ultrarunner participants may be indicated. This study included 37 interviews conducted with 28 different ultrarunners, plus field observation at two different ultramarathon events. For future study, it might be valuable to use a case-study format, in which the researchers focus on a single event and also interview event volunteers, race directors, ultrarunner family members, as well as their medical providers to gain a broader perspective on running-related injury prevention and management. Implications for Practice and Future Research RUNNING-RELATED INJURIES IN ULTRARUNNERS 91 Professional running coaches and medical professionals working with ultrarunners, should design training programs and implement rehabilitation protocols so as to facilitate the prevention and management of running-related injuries. In doing so, coaches and clinicians can help guide ultrarunners to avoid unnecessary barriers that might otherwise keep these endurance athletes from remaining healthy or achieving a swift and complete recovery from running-related injuries. Future research should consider insights into the physical and mental aspects of ultrarunning reported in this study and direct future research at identifying risk factors for running-related injuries in ultrarunners, and develop a better understanding of the social, behavioral, and psychological factors that influence prevention and management strategies for these injuries. Professional running coaches. Professional coaches should support the listen to your body strategy adopted by ultrarunners and help clients to develop skills that allow them to effectively monitor their heart rate, perceived exertion, and recovery time following running activity. Additionally, coaches and runners should be able to recognize typical and atypical responses regarding rest and activity during different phases of the training cycle and familiarize themselves with the early signs associated with adrenal fatigue and over-training syndrome. Running coaches and ultrarunners alike, should be cautious regarding tendencies of inexperienced runners to adopt strategies with pre-determined, high mileage schedules and without adequate rest and recovery, creating the potential for overuse running-related injuries. Future research should explore the experiences of endurance athletes who have struggled with adrenal fatigue and over-training syndrome in order to better understand possible contributing factors and strategies implemented for recovery of function. RUNNING-RELATED INJURIES IN ULTRARUNNERS 92 In this study, some of the interviewees described training activity that included a specific target for weekly mileage, while other ultrarunners described adhering to a system that emphasized alternative metrics including vertical gain, perceived exertion, or training time on feet. While the data collected for this study suggested a greater level of success with avoiding a mileage focused approach to training activity, future studies should investigate training activity and incidence of running-related injuries as well as metrics used to monitor training activity. Also, it would be beneficial to consider the incidence of running-related injuries comparing more complex training programs using periodization strategies versus strategies based on felt and perceived exertion. Many of the ultrarunners in this study described taking a too much, too soon approach following a positive first 50 km race experience. Ultrarunners should carefully consider their strategy for scheduling ultramarathon events and avoid overscheduling that does not allow for adequate base building and time for rest and recovery between race events. This is especially important for the prevention and management of running-related injuries. Ultrarunners characterized their behavior during event competition as focused on performance and finishing race events, altering the listen to your body and back on track strategies employed during training activity in an effort to improve event performance and to achieve the goal of finishing the race event. Many of the more experienced ultrarunners in this study described benefitting from an annual off-season, lasting weeks or months followed by a return to running activity that adhered to no more than a 10% weekly increase in mileage as a base-building strategy. Also, prevention strategies that include trail running rather than running on paved surfaces, as well as cross-training activities may be helpful for minimizing ground-reaction forces associated with running activity. RUNNING-RELATED INJURIES IN ULTRARUNNERS 93 Previous studies have identified a higher prevalence of running-related injuries for ultrarunners who are younger and less experienced with running, and those who had relatively less focus on running, spent a greater proportion of their exercise time at high intensities, and were more likely to have performed regular resistance training (Hoffman, 2016). It would be helpful to investigate the incidence of running-related injuries for ultrarunners who adhere to a designated off-season with considerably scaled back running activity and without scheduled running events. The interviewees for this study who included blocks of time off running, also reported having success with remaining free of running-related injuries. Running speed, stride frequency, and impact forces, as well as trail rather than paved surfaces, have been suggested as strategies for mitigating overuse injuries for ultramarathon runners (Hoffman & Krishnan, 2013; Knechtle, 2012). The participants in this study described variable degrees of success attempting to alter their running form for the purpose of minimizing impact during running. Nonetheless, professional coaches and physical therapists familiar with differences in ground reaction forces and loading rates associated with variations in running form may be able to offer suggestions to alter the kinetic variables associated with RRI in specific populations of distance runners. The interviewees in this study described yoga, biking, swimming, weight lifting, snow-skiing, and rock-climbing as effective way to diversify their training schedule and to include activities with reduced impact forces than those associated with the repetitive running motion. Future research should investigate the relationship between different cross-training strategies and the incidence of running-related injuries, which would also be beneficial to ultrarunners, coaches, and medical professionals working with individuals in the sport. Future research should investigate the use of RUNNING-RELATED INJURIES IN ULTRARUNNERS 94 blood flow restriction therapy for ultrarunner training activity as a strategy for reducing the mileage necessary to achieve a desired level of fitness and therefore minimize impact. Medical professionals. Rehabilitation specialists, including physical therapists should assist ultrarunners by providing the knowledge and skills necessary to help patients to implement a back on track strategy for the management of running-related injuries. While the incidence of running-related injuries reported was seemingly less than what is reported in the literature for ultrarunners, the types of injuries incurred were consistent with those described in other studies. Medical professionals can facilitate the recovery process for running-related injuries by providing accurate diagnoses and collaborating with other medical providers when appropriate in order to make a proper differential diagnosis for the injuries most common to ultrarunners including the knee, calf, hamstring, Achilles, and low back. Future research should include prospective studies conducted on the ultrarunning population to help identify the risk factors associated with running-related injuries. There is little understanding of the relationship between training activity and the incidence of overuse running-related injuries, particularly with regard to variables such as rest and recovery, vertical gain, and cross-training activities. Also, researchers should build upon the current evidence related to specific treatment interventions such as strengthening, stretching, and dry needling for running-related injuries for distance runners. While there is evidence supporting the use of blood flow restriction therapy for strength building, there are no studies that describe its use for ultrarunners (Cook et al., 2014; Loenneke & Pujol, 2009). Future research should explore the feasibility of blood flow restriction therapy to decrease weekly running distance for ultrarunners who describe weekly running distances of up to 80 miles per week. Additionally, the RUNNING-RELATED INJURIES IN ULTRARUNNERS 95 use of blood flow restriction therapy compared to other treatments for running-related injuries should be explored. The participants interviewed for this study identified overuse injuries as the most common type of injury incurred, which is consistent with the research. As such, ultrarunners are frequently presented with deciding whether or not a hold on running activity is necessary. Medical professionals can guide ultrarunners on how to interpret pain in a way that is injury specific and that aims for full recovery of function and return to running activity while also avoiding unnecessary delays in progress due to reaggravation of ongoing overuse. In particular, the management of stress fractures which were commonly reported by the ultrarunners in this study and supported in the literature should include education regarding the early signs and most common locations for stress fracture injuries for distance runners. Also, coaches and clinicians should be mindful of positive thinking strategies used by some ultrarunners that include focusing on good feeling rather than hurting areas of the body during race events. Clinicians, including physical therapists working with ultrarunners over a period of time should help these patients with pain interpretation strategies based upon specific injury diagnosis to ensure a full recovery process that avoids unnecessary delays in healing and include guidance on an appropriate level of running activity. Stress fractures were a commonly reported injury for the ultrarunners in this study, which is consistent with the findings of other authors (Hoffman, 2016; Snyder, Koester, & Dunn, 2006). The participants in this study described difficulties sorting out transient aches and pains from overuse injuries potentially worsened by running activity. Future research is needed to investigate strategies aimed at identifying the risk factors associated with stress fractures in the ultrarunner population and in assisting these athletes with the transition from the prevention and RUNNING-RELATED INJURIES IN ULTRARUNNERS 96 performance strategies to the management strategies. It may also be useful to investigate how the running form for ultrarunners compares with that of shorter distance runners and to identify variables that have been associated with running-related injuries for shorter distance runners, such as loading rate and ground reaction forces (Adler, 2013). Ultrarunners, as well as professional coaches and medical professionals should consider the tendency of the over-achiever personality type to view the prevention and management as unnecessary parts of participation in the sport. Also, coaches and clinicians are in unique position to recognize a shift away from the me against me approach that includes friendly competition, toward an approach that focuses on outside influences to be successful and with unhealthy amounts of associated stress. These challenges may be particularly poignant for high level ultrarunners with podium finish experiences. Professional coaches and medical providers should be cognizant of the difficulties associated with a me against me approach to participation in the sport given the potential pressure added by social media and the accessibility of performance tracking tools currently available. It may also be useful for researchers to apply personality profiling and mental strategies related to the associative and dissociative strategies. In particular, studies taking into consideration the effectiveness of different strategies aimed at differentiating transient aches and pains considered to be normal for ultrarunning versus significant runningrelated injuries worsened by running activity and leading to lengthy periods of time off running should be studied. The first steps to this line of inquiry may include using survey instruments to gather information related to ultrarunner personality traits, as well as tests that predict success in areas of life apart from sport participation, along with the training habits for ultrarunners. Lastly, medical professionals should make every effort to include fitness alternatives in the plan of care in situations where healing necessitates time off running. Healthcare providers RUNNING-RELATED INJURIES IN ULTRARUNNERS 97 should also be mindful that ultrarunners are accustomed to high volumes of outdoor physical activity. Therefore, it may be beneficial to provide guidance on fitness alternatives such as cycling or swimming, cross-country skiing, walking or hiking with an emphasis on the significant role of pain interpretation that is conducive to healing. Providers should be cognizant of the need to consider referral ultrarunner patients to sport psychologists or to primary care physicians if clinical depression is suspected during periods of extended time off running due to running-related injuries. Conclusion This study described how ultrarunners approach prevention and management strategies for running-related injuries. The three themes, listen to your body, back on track, and mental toughness are described in the conceptual model (Figure 1). Psychological and behavioral factors related to ultrarunning participation served as both barriers and facilitators to prevention and management strategies for running-related injuries. Study participants described experiences in which rigid adherence to high-volume training programs and over-scheduling ultramarathon events resulted in overuse running-related injuries. In contrast, training activity guided by perceived exertion, time on feet, and focusing on a particular goal race for the calendar year were described as effective injury prevention strategies. Most of the study participants reported obligatory tendencies toward training activity and described tracking tools that provide transparency for ultrarunning performance as a barrier to implementing a me against me strategy for running-related injuries. Management strategies included problem-solving to identify the nature of the injury, and to determine whether or not to continue running activity. Some ultrarunners employed positive thinking strategies to influence pain interpretation in order to finish ultramarathon events, particularly runners with ultramarathon podium or trophy finishes. RUNNING-RELATED INJURIES IN ULTRARUNNERS 98 Participants described mixed results using positive thinking strategies to manage running-related injuries. Challenges associated with growth in the sport, included perceived pressure associated with online performance tracking tools and difficulties withdrawing from race events requiring earned points or lottery selection for entrance. Ultrarunners reported negative experiences seeking care from medical professionals who were averse to ultrarunning activity. Positive experiences were reported by study participants who utilized fitness alternatives as a way to boost their physical and mental health during time off running. RUNNING-RELATED INJURIES IN ULTRARUNNERS 99 References Aldiabat, K. M., & Le Navenec, C. L. (2018). Data saturation: The mysterious step in grounded theory methodology. Qualitative Report, 23(1), 245-261. Almeida, M. O., Davis, I. S., & Lopes, A. D. (2015). Biomechanical differences of foot-strike patterns during running: A systematic review with meta-analysis. Journal of Orthopaedic & Sports Physical Therapy, 45(10), 738-755. doi:10.2519/jospt.2015.6019 Almekinders, L. C., & Engle, C. R. (2019). Common and uncommon injuries in ultra-endurance sports. Sports Medicine and Arthroscopy Review, 27(1), 25-30. Almekinders, L. C., & Engle, C. R. (2019). Common and uncommon injuries in ultra-endurance sports. Sports Medicine and Arthroscopy Review 27(1), 25-30. doi:10.1097/jsa.0000000000000217 Alschuler, K. N., Kratz, A. L., Lipman, G. S., Krabak, B. J., Pomeranz, D., Burns, P., . . . Jensen, M. P. (2019). How variability in pain and pain coping relate to pain interference during multistage ultramarathons. Pain, 160(1), 257-262. doi:10.1097/j.pain.0000000000001397 Antonini Philippe, R., Rochat, N., Vauthier, M., & Hauw, D. (2016). The story of withdrawals during an ultra-trail running race: A qualitative investigation of runners' courses of experience. Sport Psychologist, 30(4), 361-375. Balducci, P., Clmenon, M., Trama, R., Blache, Y., & Hautier, C. (2017). Performance factors in a mountain ultramarathon. International Journal of Sports Medicine, 38(11), 819-826. doi:10.1055/s-0043-112342 Bertelsen, M. L., Jensen, J. F., Nielsen, M. H., Nielsen, R. O., & Rasmussen, S. (2013). Footstrike patterns among novice runners wearing a conventional, neutral running shoe. Gait Posture, 38(2), 354-356. doi:10.1016/j.gaitpost.2012.11.022 Bossi, A. H., Matta, G. G., Millet, G. Y., Lima, P., Pertence, L. C., de Lima, J. P., & Hopker, J. G. (2017). Pacing strategy during 24-hour ultramarathon-distance running. International RUNNING-RELATED INJURIES IN ULTRARUNNERS 100 Journal of Sports Physiology & Performance, 12(5), 590-596. doi:10.1123/ijspp.20160237 Breslin, G., Murphy, M. H., Kremer, J., McClean, C., & Davison, G. (2014). Providing sport psychology support to an athlete in a unique, ultra-endurance event. Journal of Sport Psychology in Action, 5(2), 59-72. Buck, K., Spittler, J., Reed, A., & Khodaee, M. (2018). Psychological attributes of ultramarathoners. Wilderness and Environmental Medicine, 29(1), 66-71. doi:10.1016/j.wem.2017.09.003 Bhlmayer, L., Birrer, D., Rthlin, P., Faude, O., & Donath, L. (2017). The effects of mindfulness practiceson performance-relevant parameters and performance outcomes in sports: A meta-analytical review. Sports Medicine, 47(11), 2309-2321. doi:10.1007/s40279-017-0752-9 Chalabaev, A., Radel, R., Ben Mahmoud, I., Massiera, B., Deroche, T., & d'Arripe-Longueville, F. (2017). Is motivation for marathon a protective factor or a risk factor of injury? Scandinavian Journal of Medicine & Science in Sports, 27(12), 2040-2047. doi:10.1111/sms.12807 Charmaz, K. (2014). Constructing grounded theory: A practical guide through qualitative analysis (2nd ed.). London, England: SAGE Publications. Claire, B. D., Donna, S. M., Ratchneewan, R., & Thomas, B. R. (2007). Theoretical sampling and category development in grounded theory. Qualitative Health Research, 17(8), 11371148. doi:10.1177/1049732307308450 RUNNING-RELATED INJURIES IN ULTRARUNNERS 101 Cook, C. J., Kilduff, L. P., & Beaven, C. M. (2014). Improving strength and power in trained athletes with three weeks of occlusion training. International Journal of Sports Physiology and Performance, 9(1), 166-172. doi:10.1123/ijspp.2013-0018 Covassin, T., Beidler, E., Ostrowski, J., & Wallace, J. (2015). Psychosocial aspects of rehabilitation in sports. Clinical Sports Medicine, 34(2), 199-212. doi:10.1016/j.csm.2014.12.004 Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: SAGE Publications. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry & research design: Choosing among five approaches (4th ed.). Thousand Oaks, CA: SAGE Publications. Crust, L., & Azadi, K. (2010). Mental toughness and athletes' use of psychological strategies. European Journal of Sport Science, 10(1), 43-51. Damsted, C., Parner, E. T., Srensen, H., Malisoux, L., Hulme, A., & Nielsen, R. (2019). The association between changes in weekly running distance and running-related injury: Preparing for a half marathon. The Journal of Orthopaedic and Sports Physical Therapy, 49(4), 230-238. doi:10.2519/jospt.2019.8541 Davis, I., Bowser, B., & Mullineaux, D. (2016). Greater vertical impact loading in female runners with medically diagnosed injuries: A prospective investigation. British Journal of Sports Medicine, 50(14), 887-887. doi:10.1136/bjsports-2015-094579 Davis, I. S., Rice, H. M., & Wearing, S. C. (2017). Why forefoot striking in minimal shoes might positively change the course of running injuries. Journal of Sport and Health Science. 6, 154161. RUNNING-RELATED INJURIES IN ULTRARUNNERS 102 De Petrillo, L. A., Kaufman, K. A., Glass, C. R., & Arnkoff, D. B. (2009). Mindfulness for longdistance runners: An open trial using mindful sport performance enhancement (MSPE). Journal of Clinical Sport Psychology, 3(4), 357-376. doi:10.1123/jcsp.3.4.357 Degache, F., Morin, J. B., Oehen, L., Guex, K., Giardini, G., Schena, F., . . . Millet, G. P. (2016). Running mechanics during the world's most challenging mountain ultramarathon. International Journal of Sports Physiology and Performance, 11(5), 608-614. doi:10.1123/ijspp.2015-0238 DUV Ultra Marathon Statistics. (2018). Retrieved from http://statistik.d-u-v.org Esculier, J. F., Dubois, B., Dionne, C. E., Leblond, J., & Roy, J. S. (2015). A consensus definition and rating scale for minimalist shoes. Journal of Foot and Ankle Research, 8, 42. doi:10.1186/s13047-015-0094-5 Ferley, D. D., Osborn, R. W., & Vukovich, M. D. (2014). The effects of incline and level-grade high-intensity interval treadmill training on running economy and muscle power in welltrained distance runners. Journal of Strength and Conditioning Research, 28(5), 12981309. doi:10.1519/JSC.0000000000000274 Fields, K. B., Sykes, J. C., Walker, K. M., & Jackson, J. C. (2010). Prevention of running injuries. Current Sports Medicine Reports, 9(3), 176-182. doi:10.1249/JSR.0b013e3181de7ec5 Freund, W., Weber, F., Billich, C., Birklein, F., Breimhorst, M., & Schuetz, U. H. (2013). Ultramarathon runners are different: Investigations into pain tolerance and personality traits of participants of the TransEurope FootRace 2009. Pain Practice, 13(7), 524-532. doi:10.1111/papr.12039 RUNNING-RELATED INJURIES IN ULTRARUNNERS 103 Fuller, J. T., Thewlis, D., Buckley, J. D., Brown, N. A., Hamill, J., & Tsiros, M. D. (2017). Body mass and weekly training distance influence the pain and injuries experienced by runners using minimalist shoes: A randomized controlled trial. The American Journal of Sports Medicine, 45(5), 1162-1170. doi:10.1177/0363546516682497 Giandolini, M., Vernillo, G., Samozino, P., Horvais, N., Edwards, W. B., Morin, J.B., & Millet, G. Y. (2016). Fatigue associated with prolonged graded running. European Journal of Applied Physiology, 116(10), 1859-1873. doi:10.1007/s00421-016-3437 Giovanelli, N., Taboga, P., & Lazzer, S. (2017). Changes in running mechanics during a 6-hour running race. International Journal of Sports Physiology and Performance, 12(5), 642647. doi:10.1123/ijspp.2016-0135 Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968). The discovery of grounded theory: Strategies for qualitative research. Nursing Research, 17(4), 364-365. doi:10.1097/00006199196807000-00014 Gledhill, A., Forsdyke, D., & Murray, E. (2018). Psychological interventions used to reduce sports injuries: A systematic review of real-world effectiveness. British Journal of Sports Medicine, 52(15), 967-971. doi:10.1136/bjsports-2017-097694 Goss, D., & Gross, M. (2012). Relationships among self-reported shoe type, footstrike pattern, and injury incidence. U.s. Army Medical Department Journal, 25-30, 25-30. Hanson, N. J., & Buckworth, J. (2015). The effect of endpoint knowledge on perceived exertion, affect, and attentional focus during self-paced running. Journal of Strength and Conditioning Research, 29(4), 934-941. Heapy, A. M., Hoffman, M. D., Verhagen, H. H., Thompson, S. W., Dhamija, P., Sandford, F. J., & Cooper, M. C. (2018). A randomized controlled trial of manual therapy and pneumatic RUNNING-RELATED INJURIES IN ULTRARUNNERS 104 compression for recovery from prolonged running - an extended study. Research in Sports Medicine, 26(3), 354-364. doi:10.1080/15438627.2018.1447469 Henderson, R., & Rheault, W. (2004). Appraising and incorporating qualitative research in evidence-based practice. Journal of Physical Therapy Education, 18(3), 35-40. Hespanhol, L. C., Jr., van Mechelen, W., & Verhagen, E. (2018). Effectiveness of online tailored advice to prevent running-related injuries and promote preventive behavior in Dutch trail runners: A pragmatic randomized controlled trial. British Journal of Sports Medicine, 52(13), 851-858. doi:10.1136/bjsports-2016-097025 Hoffman, M., Lee, J., Zhao, H., & Tsodikov, A. (2007). Pain perception after running a 100-mile ultramarathon. Archives of Physical Medicine and Rehabilitation, 88(8), 1042-1048. Hoffman, M. (2016). Injuries and health considerations in ultramarathon runners. Physical Medicine & Rehabilitation Clinics of North America, 27(1), 203-216. Hoffman, M., Badowski, N., Chin, J., & Stuempfle, K. J. (2016). A randomized controlled trial of massage and pneumatic compression for ultramarathon recovery. Journal of Orthopaedic & Sports Physical Therapy, 46(5), 320-326. doi:10.2519/jospt.2016.6455 Hoffman, M., & Fogard, K. (2011). Factors related to successful completion of a 161-km ultramarathon. International Journal of Sports Physiology and Performance, 6(1), 25-37. Hoffman, M., & Krishnan, E. (2013). Exercise behavior of ultramarathon runners: Baseline findings from the ULTRA study. Journal of Strength and Conditioning Resource, 27(11), 2939-2945. doi:10.1519/JSC.0b013e3182a1f261 Hoffman, M., Krishnan, E., & Lucia, A. (2014). Health and exercise-related medical issues among 1,212 ultramarathon runners: Baseline findings from the Ultrarunners Longitudinal TRAcking (ULTRA) Study. PLoS ONE, 9(1), e83867. RUNNING-RELATED INJURIES IN ULTRARUNNERS 105 Hoffman, M., & Krouse, R. (2018). Ultra-obligatory running among ultramarathon runners. Research in Sports Medicine, 26(2), 211-221. doi:10.1080/15438627.2018.1431533 Hohl, R., Ferraresso, R. L. Z. P., De Oliveira, R. B., Lucco, R., Brenzikofer, R., & De Macedo, D. V. (2009). Development and characterization of an overtraining animal model. Medicine & Science in Sports & Exercise, 41(5), 1155-1163. doi:10.1249/MSS.0b013e318191259c Holt, N. L., Homan, L., Youngoh, K., & Klein, K. (2014). Exploring experiences of running an ultramarathon. Sport Psychologist, 28(1), 22-35. Ivarsson, A., Johnson, U., Andersen, M. B., Tranaeus, U., Stenling, A., & Lindwall, M. (2017). Psychosocial factors and sport injuries: Meta-analyses for prediction and prevention. The American Journal of Sports Medicine, 47(2), 353-365. doi:10.1007/s40279-016-0578-x Jaeschke, A.-M. C., Sachs, M. L., & Dieffenbach, K. D. (2016). Ultramarathon runners' perceptions of mental toughness: A qualitative inquiry. Sport Psychologist, 30(3). Janice, M. M., Michael, B., Maria, M., Karin, O., & Jude, S. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), 13-22. Jelvegrd, S., Timpka, T., Bargoria, V., Gauffin, H., & Jacobsson, J. (2016). Perception of health problems among competitive runners: A qualitative study of cognitive appraisals and behavioral responses. Orthopaedic Journal of Sports Medicine, 4(12), 2325967116673972. doi:10.1177/2325967116673972 Kellmann, M., Bertollo, M., Bosquet, L., Brink, M., Coutts, A. J., Duffield, R., . . . Beckmann, J. (2018). Recovery and performance in sport: Consensus statement. International Journal of Sports Physiology and Performance, 13(2), 240-245. doi:10.1123/ijspp.2017-0759 RUNNING-RELATED INJURIES IN ULTRARUNNERS 106 Kenneally, M., Casado, A., & Santos-Concejero, J. (2017). The effect of periodisation and training intensity distribution on middle- and long-distance running performance: A systematic review. International Journal of Sports Physiology and Performance, 1-26. doi:10.1123/ijspp.2017-0327 Kim, T. H., Lee, M. S., Choi, T. Y., Kim, K. H., Kang, J. W., & Ernst, E. (2014). Acupuncture for treating acute ankle sprains in adults. Cochrane Database of Systematic Reviews, 2014(6). doi:10.1002/14651858.CD009065.pub2 Kimmerle, J., Gerbing, K.-K., Thiel, A., & Cress, U. (2012). Exchange of complementary and alternative medical knowledge in sport-related internet fora. Sociology of Sport Journal, 29(3), 348-364. Kluitenberg, B., van der Worp, H., Huisstede, B. M. A., Hartgens, F., Diercks, R., Verhagen, E., & van Middelkoop, M. (2016). The NLstart2run study: Training-related factors associated with running-related injuries in novice runners. Journal of Science and Medicine in Sport, 19(8), 642-646. Kluitenberg, B., van Middelkoop, M., Diercks, R., & van der Worp, H. (2015). What are the differences in injury proportions between different populations of runners? A systematic review and meta-analysis. Sports Medicine, 45(8), 1143-1161. doi:10.1007/s40279-0150331-x Knapik, J. J., Swedler, D. I., Grier, T. L., Hauret, K. G., Bullock, S. H., Williams, K. W., . . . Jones, B. H. (2009). Injury reduction effectiveness of selecting running shoes based on plantar shape. Journal of Strength and Conditioning Research, 23(3), 685-697. Knechtle, B. (2012). Ultramarathon runners: Nature or nurture? International Journal of Sports Physiology and Performance, 7(4), 310-312. RUNNING-RELATED INJURIES IN ULTRARUNNERS 107 Knechtle, B., Rosemann, T., Zingg, M. A., Stiefel, M., & Rst, C. A. (2015). Pacing strategy in male elite and age group 100 km ultra-marathoners. Open Access Journal of Sports Medicine, 6, 71-80. doi:10.2147/OAJSM.S79568 Kreher, J. B., & Schwartz, J. B. (2012). Overtraining syndrome: A practical guide. Sports Health, 4(2), 128-138. Kozinc, ., & arabon, N. (2017). Effectiveness of movement therapy interventions and training modifications for preventing running injuries: A meta-analysis of randomized controlled trials. Journal of Sports Science & Medicine, 16(3), 421-428. Krouse, R. Z., Ransdell, L. B., Lucas, S. M., & Pritchard, M. E. (2011). Motivation, goal orientation, coaching, and training habits of women ultrarunners. The Journal of Strength & Conditioning Research, 25(10), 2835-2842. doi:10.1519/JSC.0b013e318204caa0 Le Meur, Y., Louis, J., Aubry, A., Guneron, J., Pichon, A., Schaal, K., . . . Hausswirth, C. (2014). Maximal exercise limitation in functionally overreached triathletes: Role of cardiac adrenergic stimulation. Journal Of Applied Physiology (Bethesda, Md.: 1985), 117(3), 214-222. doi:10.1152/japplphysiol.00191.2014 Lentino CV, Purvis DL, Murphy KJ, Deuster PA. (2013). Sleep as a component of the performance triad: the importance of sleep in a military population.US Army Medical Department Journal, Oct.Dec. 2013, 98108. Loenneke, J. P., & Pujol, T. J. (2009). The use of occlusion training to produce muscle hypertrophy. Strength and Conditioning Journal, 31(3), 77-84. doi:10.1519/SSC.0b013e3181a5a352 Main, L. C., Dawson, B., Heel, K., Grove, J. R., Landers, G. J., & Goodman, C. (2010). Relationship between inflammatory cytokines and self-report measures of training RUNNING-RELATED INJURIES IN ULTRARUNNERS 108 overload. Research in Sports Medicine, 18(2), 127-139. doi:10.1080/15438621003627133 Malisoux, L., Chambon, N., Delattre, N., Gueguen, N., Urhausen, A., & Theisen, D. (2016). Injury risk in runners using standard or motion control shoes: A randomised controlled trial with participant and assessor blinding. British Journal of Sports Medicine, 50(8), 481. Retrieved from doi:10.1136/bjsports-2015-095031 Marshall, C., & Rossman, G. B. (2011). Designing qualitative research (5th ed.). London: SAGE Publications. Mason, D. L., Dickens, V. A., & Vail, A. (2012). Rehabilitation for hamstring injuries. Cochrane Database of Systematic Reviews(12). doi:10.1002/14651858.CD004575.pub3 Martin, R. L., Davenport, T. E., Reischl, S. F., McPoil, T. G., Matheson, J. W., Wukich, D. K., . . . Godges, J. J. (2014). Heel pain - plantar fasciitis: Revision 2014. Journal of Orthopaedic & Sports Physical Therapy, 44(11), A1-A33. doi:10.2519/jospt.2014.0303 Martin, T., Arnal, P. J., Hoffman, M. D., & Millet, G. Y. (2018). Sleep habits and strategies of ultramarathon runners. PLoS ONE, 13(5), e0194705-e0194705. doi:10.1371/journal.pone.0194705 Meeusen, R., Duclos, M., Foster, C., Fry, A., Gleeson, M., Nieman, D., . . . Urhausen, A. (2013). Prevention, diagnosis, and treatment of the overtraining syndrome: Joint consensus statement of the European College of Sport Science and the American College of Sports Medicine. Medicine and Science in Sports and Exercise, 45(1), 186-205. doi:10.1249/MSS.0b013e318279a10a Micklesfield, L. K., Hugo, J., Johnson, C., Noakes, T. D., & Lambert, E. V. (2007). Factors associated with menstrual dysfunction and self-reported bone stress injuries in female RUNNING-RELATED INJURIES IN ULTRARUNNERS 109 runners in the ultra- and half-marathons of the Two Oceans. British Journal of Sports Medicine, 41(10), 679-683. doi:10.1136/bjsm.2007.037077 Morgan, W. P., & Pollock, M. L. (1977). Psychologic characterization of the elite distance runner. Annals of New York Academy of Sciences, 301, 382-403. Mucha, M. D., Caldwell, W., Schlueter, E. L., Walters, C., & Hassen, A. (2017). Hip abductor strength and lower extremity running related injury in distance runners: A systematic review. Journal of Science and Medicine in Sport, 20(4), 349-355. doi:10.1016/j.jsams.2016.09.002 Multhaupt, G., & Beuth, J. (2018). The use of imagery in athletic injury rehabilitation: A systematic review. The German Journal of Sports Medicine, 69(3), 57-64. Nieves, J. W., Melsop, K., Curtis, M., Kelsey, J. L., Bachrach, L. K., Greendale, G., . . . Sainani, K. L. (2010). Nutritional factors that influence change in bone density and stress fracture risk among young female cross-country runners. Physical Medicine and Rehabilitation, 2(8), 740-750; quiz 794. doi:10.1016/j.pmrj.2010.04.020 Ogles, B. M., & Masters, K. S. (2003). A typology of marathon runners based on cluster analysis of motivations. Journal of Sport Behavior, 26(1). Padgett, D. K. (2012). Qualitative and mixed methods in public health. Los Angeles, CA: SAGE. Paton, C. D., Addis, S. M., & Taylor, L.-A. (2017). The effects of muscle blood flow restriction during running training on measures of aerobic capacity and run time to exhaustion. European Journal of Applied Physiology, 117(12), 2579-2585. doi:10.1007/s00421-0173745-3 Pereira, B. C., Pauli, J. R., De Souza, C. T., Ropelle, E. R., Cintra, D. E., Freitas, E. C., & da Silva, A. S. R. (2014). Eccentric exercise leads to performance decrease and insulin RUNNING-RELATED INJURIES IN ULTRARUNNERS 110 signaling impairment. Medicine and Science in Sports and Exercise, 46(4), 686-694. doi:10.1249/MSS.0000000000000149 Philippe, R. A., & Seiler, R. (2005). Sex differences on use of associative and dissociative cognitive strategies among male and female athletes. Perceptual and Motor Skills, 101(2), 440-444. Pruitt, K. A., & Hill, J. M. (2017). Optimal pacing and carbohydrate intake strategies for ultramarathons. European Journal of Applied Physiology, 117(12), 2527-2545. doi:10.1007/s00421-017-3741-7 Ristolainen, L., Kettunen, J. A., Waller, B., Heinonen, A., & Kujala, U. M. (2014). Trainingrelated risk factors in the etiology of overuse injuries in endurance sports. The Journal of sports medicine and physical fitness, 54(1), 78-87. Rochat, N., Hauw, D., Gr, G., & Seifert, L. (2018). Understanding trail runners' activity on online community forums: An inductive analysis of discussion topics. Journal of Human Kinetics, 61(1), 263-276. Roebuck, G. S., Fitzgerald, P. B., Urquhart, D. M., Ng, S.-K., Cicuttini, F. M., & Fitzgibbon, B. M. (2018). The psychology of ultra-marathon runners: A systematic review. Psychology of Sport and Exercise, 37, 43-58. doi:https://doi.org/10.1016/j.psychsport.2018.04.004 Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., . . . Jinks, C. (2017). Saturation in qualitative research: exploring its conceptualization and operationalization. Retrieved from http://eresearch.qmu.ac.uk/5265/1/5265.pdf RUNNING-RELATED INJURIES IN ULTRARUNNERS 111 Schumacher, J. M., Becker, A. J., & Wiersma, L. D. (2016). Forging ahead: An examination of the experiences and coping mechanisms of channel swimmers. Sport Psychologist, 30(4), 327-338. Silver, C., & Lewins, A. (2014). Using software in qualitative research: A step-by-step guide (2nd ed.). London, England; Los Angeles, CA: SAGE Publications. Simpson, D., Post, P. G., Young, G., & Jensen, P. R. (2014). "It's not about taking the easy road": The experiences of ultramarathon runners. Sport Psychologist, 28(2), 176-185. Sinclair, J., Fau-Goodwin, J., Richards, J., & Shore, H. (2016). The influence of minimalist and maximalist footwear on the kinetics and kinematics of running. Footwear Science, 8(1), 33-39. doi:10.1080/19424280.2016.1142003 Smith, L. L. (2000). Cytokine hypothesis of overtraining: A physiological adaptation to excessive stress? Medicine and Science in Sports and Exercise, 32(2), 317-331. Snyder, R. A., Koester, M. C., & Dunn, W. R. (2006). Epidemiology of stress fractures. Clinics in Sports Medicine, 25(1), 37-52. doi:https://doi.org/10.1016/j.csm.2005.08.005 Strauss, A. L., & Corbin, J. M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (Vol. 2). London, England: SAGE Publications. Stuckey, H. (2015). The second step in data analysis: Coding qualitative research data. Journal of Social Health and Diabetes, 3(1), 7-10. Sugimoto, D., Myer, G. D., Foss, K. D. B., & Hewett, T. E. (2015). Specific exercise effects of preventive neuromuscular training intervention on anterior cruciate ligament injury risk reduction in young females: Meta-analysis and subgroup analysis. British Journal of Sports Medicine, 49(5), 282-289. doi:10.1136/bjsports-2014-093461 RUNNING-RELATED INJURIES IN ULTRARUNNERS 112 Tan, P. L., Tan, F. H., & Bosch, A. N. (2016a). Similarities and differences in pacing patterns in a 161-km and 101-km ultra-distance road race. Journal of Strength and Conditioning Research, 30(8), 2145-2155. doi:10.1519/jsc.0000000000001326 Tan, G., Teo, I., Anderson, K. O., & Jensen, M. P. (2011). Adaptive versus maladaptive coping and beliefs and their relation to chronic pain adjustment. The Clinical Journal of Pain, 27(9), 769-774. doi:10.1097/AJP.0b013e31821d8f5a UltraRunning Finishes - U.S. (2018). Retrieved from https://calendar.ultrarunning.com/stats/ultrarunning-finishes?distance=&country=USA United States Road Race Trends. (2018). Retrieved from http://www.runningusa.org/statistics van der Worp, M. P., Haaf, D. S. T., Cingel, R. v., Wijer, A. d., Nijhuis-Van der Sanden, M. W. G., & Staal, J. B. (2015). Injuries in runners: A systematic review on risk factors and sex differences. PLoS ONE, 10. doi: 10.1371/journal.pone.0114937 van Gent, R. N., Siem, D., van Middelkoop, M., van Os, A. G., Bierma-Zeinstra, S. M., & Koes, B. W. (2007). Incidence and determinants of lower extremity running injuries in long distance runners: A systematic review. British Journal of Sports Medicine, 41(8), 469480. Videbk, S., Bueno, A. M., Nielsen, R. O., & Rasmussen, S. (2015). Incidence of runningrelated injuries per 1000 h of running in different types of runners: A systematic review and meta-analysis. Sports Medicine, 45(7), 1017-1026. doi:10.1007/s40279-015-0333-8 Wardenaar, F. C., Hoogervorst, D., Versteegen, J. J., van der Burg, N., Lambrechtse, K. J., & Bongers, C. (2018). Real-time observations of food and fluid timing during a 120 km ultramarathon. Frontiers in Nutrition, 5, 32. doi:10.3389/fnut.2018.00032 RUNNING-RELATED INJURIES IN ULTRARUNNERS 113 Wells, M. D., Dickin, D. C., Popp, J., & Wang, H. (2018). Effect of downhill running grade on lower extremity loading in female distance runners. Sports Biomechanics, 1-9. doi:10.1080/14763141.2018.1510538 Wouters, I., Almonroeder, T., Dejarlais, B., Laack, A., Willson, J. D., & Kernozek, T. W. (2012). Effects of a movement training program on hip and knee joint frontal plane running mechanics. International Journal Of Sports Physical Therapy, 7(6), 637-646. Yamato, T. P., Saragiotto, B. T., & Lopes, A. D. (2015). A consensus definition of runningrelated injury in recreational runners: A modified Delphi approach. Journal of Orthopaedic and Sports Physical Therapy, 45(5), 375-380. doi:10.2519/jospt.2015.5741 Yeung, E. W., & Yeung, S. S. (2001). A systematic review of interventions to prevent lower limb soft tissue running injuries. British Journal of Sports Medicine, 35(6), 383-389. Yoo, J. H., Lim, B. O., Ha, M., Lee, S. W., Oh, S. J., Lee, Y. S., & Kim, J. G. (2010). A metaanalysis of the effect of neuromuscular training on the prevention of the anterior cruciate ligament injury in female athletes. Knee Surgery, Sports Traumatology, Arthroscopy: Official Journal Of The ESSKA, 18(6), 824-830. doi:10.1007/s00167-009-0901-2 Yow, B. G., Tennent, D. J., Dowd, T. C., Loenneke, J. P., & Owens, J. G. (2018). Blood flow restriction training after Achilles tendon rupture. The Journal Of Foot And Ankle Surgery: Official Publication Of The American College Of Foot And Ankle Surgeons, 57(3), 635-638. doi:10.1053/j.jfas.2017.11.0 RUNNING-RELATED INJURIES IN ULTRARUNNERS Table 1 ID S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 S22 S23 S24 S25 S26 S27 S28 114 Participant Characteristics and Ultrarunning Experience Age Gender 47 24 31 47 21 50 39 39 39 51 48 27 43 51 50 31 33 40 33 37 45 28 25 59 39 47 30 45 M F M F M M M F M M M M F M M F M M M M M M F M F F F F State Res. CO HI CO CO CO CO CO CO CO CA CO CO MO CO NV ID CO CO CO CO CO CO MT CO UT CO CA CO 50K/50Mi. 12 6 10 25 1 5 4 5 20 8 22 5 9 10 10 9 4 4 2 18 7 5 5 10 5 3 1 10 100K/100 Mi. 22 2 4 5 1 0 1 0 10 7 9 0 14 3 3 1 1 1 0 23 7 3 0 0 0 0 0 0 Timed Trophy *Incidence Event Years Finishes RRI UR 2 21+ 2 < 50% 8 1-5 3 0 6-10 0 < 50% 1 11-15 2 1 1-5 0 < 50% 0 1-5 1 0 11-15 0 None 1 1-5 0 2 16-20 2 < 50% 0 1-5 0 0 6-10 0 < 50% 0 1-5 0 0 1-5 6 < 50% 0 6-10 0 0 16-20 0 50% 2 1-5 0 1 1-5 0 < 50% 1 6-10 0 1 1-5 0 < 50% 5 11-15 0 0 1-5 0 None 0 1-5 0 0 1-5 3 None 0 16-20 0 0 1-5 0 < 50% 0 1-5 0 0 1-5 0 < 50% 0 11-15 5 Note. ID = patient identifier, K = kilometers, Mi. = miles, Res. = residence, UR = ultrarunning, *Incidence RRI = reported incidence running-related injury *Study participants were issued a survey and asked to use a particular definition of a runningrelated injury (Yamato, Saragiotto, & Lopes, 2015) to answer the following question: Which of the following statements best describes your experience with ultrarunning and running-related injuries? a) I have experienced at least one running-related injury most (50% or more) of the years Ive participated in ultrarunning; b) I have experienced a running related injury less than half of the years Ive participated in ultrarunning; c) I have not experienced any running-related injuries since I began ultrarunning? RUNNING-RELATED INJURIES IN ULTRARUNNERS 115 Theme Table Prevention Strategies for Running-Related Injuries Table 2 Theme - Listen to Your Body Category Sub-theme Runningrelated Injury Prevention Strategies Resources for Training Activity Sample Quotation So, for my last race, I trained no more than fifty miles a week. I really dialed back and I had my best race ever. It was a slightly friendlier approach. It's not about pounding my legs anymore, because I've had injuries and I've learned to be mindful of my body (Ultrarunner S10). Rest and Recovery I think another thing that just comes with experience is learning about yourself. Like, if Im debating and considering not running today, is it because I'm just being a big baby? Or am I debating because I really need the time off and that's just what I need to stay healthy (Ultrarunner S8)? Event Specific Training Its never ever about pace for me. Its more about generally spending the right amount of time on my feet and also race specific terrain. For example, if Im preparing for race that involves 30,000 feet of climbing in the mountains, Im going to be doing terrain that represents the right amount of feet per mile (Ultrarunner S1). Reduced Impact I definitely feel more beat up after I run fourteen miles on concrete than I do running fourteen miles in the mountains. I'm doing almost the exact same foot strike every single time on the roads, whereas in the mountains, there may be a big hill and I'll power hike it. If I'm going downhill, Ill totally to change up my gait. If it's rocky, things are changing. Things are changing all the time (Ultrarunner S11). Event Selection I've seen so many people go from, say a half marathon, then a 50 k and then within their first year they're already signed up for a hundred miler. And more often than not, those same people end up injured. And then it's just this chronic series of injury after injury. You really have to build the base. People don't realize how critical that part is (Ultrarunner S3). Running Coach If you're just getting into the sport, I think it's valuable to get a coach for some portion of your training and probably RUNNING-RELATED INJURIES IN ULTRARUNNERS RunningRelated Injury Prevention 116 the very beginning to say, here are the big things to pay attention to. Here's a typical training schedule that a beginner should get on. Um, you know, because it can get costly, right? So, but if you can make the initial investment to get some guidance, that's really good (Ultrarunner S7). Running Community A high point that comes to mind for running career would be the time I was signed up for a particular ultrarunning. I had spent 20 minutes packing my pack the night before and I got to the race and realized I forgot it. That's like forgetting your shoes. You don't do that. How do you forget your pack? And I had a couple of friends come to my rescue one, it gave me one of his flasks, like a 22ounce flask that I just carried with me and another friend gave me some bars. And his sister literally gave me the shirt off her back. And um, I ended up doing fine (Ultrarunner S8). I think I started out reading Ultrarunning Magazine, but then you read those magazines for a year or two and then its pretty much rehashing the same information. So, it was helpful in the beginning, but not so much after thatI say that because I developed enough connections in the community, that if I had questions or issues I would just direct them to somebody I knew. I would ask people I knew rather than look for the information on a social media site and ask the question (Ultrarunner S4). Media and Publications Performance Tracking Tools A key part of it is just ditching the ego and listening to your body and knowing when to say when. Otherwise youre going to create more issues down the road. People get a training plan and all this stuff and they feel like theyve got to follow the spreadsheet no matter what and I think that gets a lot of people in trouble. I think just with social media influences and what they're seeing on, on Facebook or Instagram, they'll do whatever it takes (Ultrarunner S18). Interviewer R1: Do you track your mileage or vertical gain as part of your training? Ultrarunner S28: No, I dont. Sometimes it's a mental thing. I get really defeated if I see my miles are down, so I dont (track my miles) much. I know some runners get caught up looking at other runners on Strava and I just RUNNING-RELATED INJURIES IN ULTRARUNNERS don't do it. I think I did back when I was more competitive, but now I'm just happy to be running. Exercise Routines I have a routine that I do, which involves, eccentric calf raises and quad stretches, and I've been very consistent with the rolling (Ultrarunner S28). 117 RUNNING-RELATED INJURIES IN ULTRARUNNERS 118 Theme Table Management Strategies for Running-Related Injuries Table 3 Theme - Back on Track Category Sub-theme RunningRelated Injury Management Strategies Sample Quotations Pain Interpretation So, when I first started running, it was a rookie mistake. Uh, I had a stress fracture in the third metatarsal on my right foot and so that's the only time I've really had to take, you know time, like 12 to 18 weeks. Just because I kept running on it and it just got worse you know (Ultrarunner S13). Injury Categories Ill get tight in my lower back with tight calves, tight hamstring, tight back, stiff, thoracic, all those thingsor if I'm getting really tired, I might roll my ankle and then that will impact me for a couple of days (Ultrarunner S7). Time Off Running I had a real low point in general, near the time of that injury when I did not want to do anything and I just moped around for like four days. Um, that to me was a signal. I was pretty, pretty well at a low point because I didn't want to do anything. I didn't even want to talk to anybody. I just completely wanted to withdrawI was able to get back to running fairly quickly. I had to, to dig myself out of that hole a little bit. Um, that was pretty icky (Ultrarunner S8). Fitness Alternatives I had to stop running probably for almost like a couple of months because of plantar fasciitis. I stopped running and I needed to find something else to do, so I started swimming and biking. Okay, great. Uh, so that's something that I just needed to do something with myself and that's why I started swimming. I didn't have any pressure on my foot like I did when I was biking, so I felt better (Ultrarunner S27). Event Participation Decisions I realized that it (the ankle) just was not ready to race basically. That was two weeks before when my race was supposed to happen. And that was basically when I decided to pull the plug and say, you know, okay, this race isn't going to happen for me this year (Ultrarunner S3). Resources Medical for Running- Professionals related Injury Management I have a torn labrum in my hip that acts up sometimes, but I have a chiropractor who does a really awesome job with it and she works on me. And usually after a week, I'm back to running and knock on wood, it's been good this past training cycle (Ultrarunner S28). RUNNING-RELATED INJURIES IN ULTRARUNNERS 119 Volunteer Activity There's probably going to be 25 of us driving up to Moab (Utah) from Colorado Springs (Colorado). It's a huge commute. Some are running, some are volunteering, a couple are injured. Its a time for everybody to support everybody (Ultrarunner S26). Medicinal Treatment I've used different CBD ointments and oils. I think if you really want to see the true benefits of the CBD, then your diet has to help out too, in terms of inflammation. Because if you're doing all the CBDs or any kind of anti-inflammatory, like turmeric you should probably think about the diet too (Ultrarunner S10). Shoe Wear I have a shoe sponsor and theres a particular brand that I love and unfortunately was acquired by another company a few years ago. If you go back to my closet, Ive got stacks of that particular shoe. When they got bought out. I was like Ive got to find that shoe because I can take that shoe out of the box and I can run a hundred miles. No blisters, nothing. I dont need the newest version. I get irritated when shoe companies change (Ultrarunner S1). RUNNING-RELATED INJURIES IN ULTRARUNNERS 120 Theme Table Ultrarunning Participation Table 4 Theme - Mental Toughness Category Sub-theme Sample Quotation Ultrarunning Personality Participation Profile Psychological Factors I think, you know, you could have an acute injury and you know, limited or walk whatever you need to do. So, I think that's a totally different story but it would probably be detrimental to you getting to that starting line if you push it too far through an injury two weeks before, three weeks before. The goal, if you really want to run the race, is to get to the starting line in as good a shape as possible, to be as prepared as you can be right now. That being said, I think a lot of people, you know, like that feeling of pushing themselves and getting far. And I think a lot of ultrarunners are probably very task oriented (Ultrarunner S6). Source of Motivation I think so much of ultrarunning is mental. I really do. Two decades ago I dont know if I would have said that. Anyway, you train your body, and you train your mind to be uncomfortable. You know that might include cold, wet, hungry, whatever. Youve got to train your mind to know this is doable (Ultrarunner S1). Pain and Suffering So, my philosophy is to just run through the pain. Pain and suffering are not long-lasting. It is part of the journey of running, and its almost an addiction and part of the sport I like. The suffering can be like a fun game for myself. How much more can I go? It depends on what kind of pain youre talking about. If its the kind of pain I know will end up causing injury to myself I will stop (Ultrarunner S2). Ultrarunning Positive Participation Thinking Behavioral Factors Let's say my feet are hurting really badIll tell myself, I'm so thankful that my knees feel good right now. Im so thankful that my back feels great right now. I will tell myself the back is good and I dont think about the feet, when in reality my feet are screaming. Thats just a little tidbit or trick that I do. My knees are good. My back is good. My stomach feels good. I have decent energy or whatever. I dont know if you would characterize that as mindfulness or what you would call that, but there is definitely a mindset of focusing on whats working (Ultrarunner S1). ProblemSolving I mean, it's kind of like a chess match. Not only do you have to be physically fit and put in the training, but there just so many variables that go into ultrarunning (Ultrarunner S3). RUNNING-RELATED INJURIES IN ULTRARUNNERS 121 It's not like running a 5K where the pain over time graph is linear. When you run ultras there are going to be more peaks and valleys. So, you're going to have to kind of ride out the lows, and know that you're going to come out of it, especially if the race is a longer distance. It's a lot about problem-solving (Ultrarunner S22). Goal Setting For me it was about a means to an end. There is the short term, but you have to remember youve got this journey and this is an ultramarathon. Youve got to remember the end goal. Its just four months. What you want is to run again (Ultrarunner S2). Running head: RUNNING-RELATED INJURIES IN ULTRARUNNERS 122 Figure 1. Grounded Theory Model: Prevention and Management of Running-Related Injuries in Ultrarunners. This figure illustrates prevention and management strategies utilized by ultrarunners for running-related injuries. Resources utilized by ultrarunners and the influence of ultrarunning participation factors are also depicted. Running head: RUNNING-RELATED INJURIES IN ULTRARUNNERS 123 Appendix A Research Study Recruitment Document (Informational Flyer) Are You an Ultrarunner? Volunteers Needed for Research Study. The purpose of the research study is to describe how ultrarunners approach prevention and management strategies for running-related injuries. Participants do not necessarily need to have experienced running-related injuries to participate in this study. You must be at least 18 years of age and have completed at least one ultramarathon in the last three years to be considered for the study.An interview, lasting 45 to 60-minutes will be conducted with study participants in either a face-to-face, public-private setting (library, or local university setting) or remotely using a virtual platform (Facetime, Skype, etc.). Participants may be asked to participate in an elective second, follow-up interview. This study has been approved by the University of Indianapolis Institutional Review Board (UIndy Study #01002, approved January 3, 2019). If you have any questions or would like to participate in the research, contact Josh White at (720) 284-1671 (text or phone) or whitej@uindy.edu. RUNNING-RELATED INJURIES IN ULTRARUNNERS 124 Appendix B Research Study Recruitment Document (Email Invitation) Hello, my name is Josh White. I am a doctoral student at University of Indianapolis in College of Health Sciences. I am conducting research on running-related injuries in ultrarunners. The purpose of the research study is to describe how ultrarunners approach prevention and management strategies for running-related injuries. I am inviting you to participate because you are at least 18 years of age and have completed at least one ultramarathon in the last three years. You do not necessarily need to have experienced running-related injuries to participate in this study. An interview, lasting 45 to 60-minutes will be conducted with study participants in either a faceto-face, public-private setting (library, or local university setting) or remotely using a virtual platform (Facetime, Skype, etc.). Participants may be asked to participate in an elective second, follow-up interview. This study has been approved by the University of Indianapolis Institutional Review Board (UIndy Study #01002, approved January 3, 2019). If you have any questions or would like to participate in the research, I can be reached at (720) 284-1671 (text or phone) or whitej@uindy.edu. RUNNING-RELATED INJURIES IN ULTRARUNNERS 125 Appendix C Basic Study Participant Information 1. What is your age? 2. How many years of running experience do you have? 3. How many years ago did you complete your first ultramarathon? 4. How many ultramarathons have you completed? 5. How many of the ultramarathon events, in which you have participated have been the 50mile distance or less? 6. How many of the ultramarathon events, in which you have participated have been greater than the 50-mile distance? 7. For the purpose of this study, a running-related (involving training or competition) injury is a musculoskeletal pain that causes a restriction or stoppage of running (including distance, speed, or duration) for at least 7 days or 3 consecutive scheduled training sessions, or that requires the runner to consult a physician or other health professional (Yamato, Saragiotto, & Lopes, 2015). Using this definition, which of the following statements best describes your experience with ultrarunning and running-related injuries: a. I have experienced at least one running-related injury most (50% or more) of the years Ive participated in ultrarunning. b. I have experienced a running related injury less than half of the years Ive participated in ultrarunning. c. I have not experienced any running-related injuries since I began ultrarunning? RUNNING-RELATED INJURIES IN ULTRARUNNERS 126 Appendix D Running-Related Injuries in Ultrarunners: Interview Guide My name is Josh White, and I am a student at the University of Indianapolis studying to earn a Doctorate of Health Science degree. I will be conducting this interview for my doctoral project. The purpose of this interview is to broaden the understanding of ultrarunner experiences and to explore the perceptions of those factors that may influence the prevention and management of running-related injuries. You will provide the perspective of an ultra-distance runner (ultrarunner) who has participated in at least one race event in which participants run further than the marathon distance of 42.2 km (26.2 mi). The responses you provide for this interview will be used for the purposes of this doctoral project study; your responses are confidential; and, any identifying information will be removed from the transcription of the audio recording. The time frame for this interview will be approximately 45-60 minutes. Your participation is voluntary, and you may choose not to answer one or more questions. The interview can be terminated at any time upon your request. Do you have any questions about the interview or how the data will be used? To ensure I have all of your responses correct, I would like to audio record the interview. The recording will be deleted once it has been utilized for the purposes of the doctoral project. Interview Guide 1. Please tell me a little about your running background and experience with ultrarunning? a. What motivates you to participate in ultrarunning? b. Tell me about a typical training week for you, say three months before your last ultra-distance running event? c. What sort of strategies have you used to contend with the unique challenges presented in ultrarunning (during training or events)? RUNNING-RELATED INJURIES IN ULTRARUNNERS 127 2. Can you describe a particular experience (or period of time) when you were faced with a running injury (either a sudden onset such as an ankle sprain, or a gradual onset such as plantar fasciitis)? a. What sort of strategies have you used for staying healthy (injury prevention)? b. How about for dealing with an existing injury (injury management)? 3. How do you make decisions about continuing to run, scaling back (altering), or discontinuing your training activity? How about race event scheduling and participation? a. Do you have any experiences with coaches (or other resources) that relate to achieving consistent injury-free training? 4. What resources have been helpful for you in regards to prevention or management strategies for running-related injuries? a. Do you have any experiences with medical professionals (or other resources) that relate to injury management strategies? 5. How would you describe the ultrarunning community? a. Can you share any examples of how the ultrarunning community has shaped how you think about or make decisions related to running-related injuries? b. In your experience, how is information related to prevention or management strategies circulated or shared amongst ultrarunners? 6. What are your thoughts on pain and suffering experienced during ultrarunning? a. Is there a difference between pain and suffering? If so, can you tell me about that distinction? RUNNING-RELATED INJURIES IN ULTRARUNNERS 128 b. What is the process by which you sort out pain/suffering that leads to improved performance (a positive outcome) versus pain/suffering that leads to injury or overtraining (a negative outcome)? 7. Do you have any ideas about or experiences using mental techniques during ultrarunning (for example, self-talk, imagery, or mindfulness practice)? a. How do (might) these practices help you to run long distances? b. Do you have any thoughts about or experiences using these mental techniques with preventing or managing running-related injuries? 8. How would you define mental toughness as it relates to ultrarunning? a. Is there a particular low point in your ultrarunning experience (career) that you feel comfortable sharing? Any that relate to challenges with injuries? b. Do you recall a high point youve experienced in ultrarunning? Any that relate to preventing or overcoming a running-related injury? 9. Are there any key lessons youve learned regarding running-related injuries and ultrarunning? a. What advice would you give to an ultrarunner who is considering injury prevention strategies? How about for someone dealing with a running-related injury challenge? 10. Is there anything else that you would like to share on this topic? Thank you for participating in the interview today. RUNNING-RELATED INJURIES IN ULTRARUNNERS 129 Appendix E Interview Guide for Ultramarathon Event 1. Tell me about your ultrarunning event experience today? a. What motivated you to sign-up for this particular ultra-distance running event? b. Tell me about your training approach, leading up to todays event? 2. Did you have any concerns regarding running-related injuries either coming into todays race as a result of training activity or during the race itself (either a sudden onset such as an ankle sprain, or a gradual onset such as plantar fasciitis)? a. What sort of strategies have you used for staying healthy (injury prevention)? b. How about for dealing with an existing injury (injury management)? 3. Did you have any unique challenges to contend with during your training or while running today? a. Were you faced with any decisions about continuing to run, scaling back (altering), or discontinuing your training activity? b. How about during todays race event? 4. Do you recall seeking out any information related to prevention or management strategies in preparation for todays event? a. Did you utilize a running coach (or other resources) to prepare for this ultradistance running event? b. Did you seek out any medical professionals (or other resources) that relate to injury management strategies? 5. Can you think of any examples of how the ultrarunning community might have impacted your event experience today? RUNNING-RELATED INJURIES IN ULTRARUNNERS 130 a. Do you have any thoughts on how the ultrarunning community might have influenced your approach toward training activity leading up to this race event? b. How about with regard to dealing with any experience running-related injuries or concerns regarding potential running-related injuries? 6. What are your thoughts on pain and suffering experienced during this ultramarathon? a. Is there a difference between pain and suffering? If so, can you tell me about that distinction? b. What is the process by which you sort out pain/suffering that leads to improved performance (a positive outcome) versus pain/suffering that leads to injury or overtraining (a negative outcome)? 7. Did you utilize any particular mental techniques during ultrarunning (for example selftalk, imagery, or mindfulness practice)? a. If so, how did this help you with your running today? b. Did you use any of these mental techniques as a part of preventing or managing any actual or potential running-related injuries today? c. How about during training activity leading up to todays event? 8. How would you define mental toughness as it relates to ultrarunning? a. Was there a particular low point in your ultrarunning experience (today) that you feel comfortable sharing? Any challenges with injuries? b. What was the high point of todays event? Did you have to overcome any particular physical barriers as a part of completing the race (could be a runningrelated injury)? RUNNING-RELATED INJURIES IN ULTRARUNNERS 131 9. Are there any key lessons you learned today? Any related to strategies related to injury prevention or injury management? a. What advice would you give to an ultrarunner who is considering this particular ultra-distance running event? 10. Is there anything else that you would like to share related to your running experience today or this topic? Thank you for participating in this interview today.  ...

... FACTORS ASSOCIATED WITH FIRST-TIME PASS RATES ON THE ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Wade M Welton, MS, LAT, ATC Copyright August 23, 2019 By: Wade M Welton, MS, LAT, ATC All rights reserved Approved by: Christine A. Lauber, EdD, LAT, ATC Committee Chair Elizabeth S. Moore, PhD Committee Member Herbert K. Amato, DA, AT-Ret Committee Member Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis Running head: ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION Factors Associated with First-Time Pass Rates on the Athletic Trainers Board of Certification Examination Wade Welton University of Indianapolis 1 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 2 Table of Contents Abstract ........................................................................................................................................... 4 Acknowledgement .......................................................................................................................... 5 Problem Statement ...................................................................................................................... 6 Study Purpose.............................................................................................................................. 7 Literature Review............................................................................................................................ 8 Historical Perspective of Athletic Training Education ............................................................. 10 Historical Perspective of Amato-Cole Educational Services Preparatory Workshop............... 13 Nursing and Predicting First-Time Board Exam Pass Rate ...................................................... 14 Predictors of First-Time Pass Rate on the National Physical Therapy Examination ................ 16 Occupational Therapy and Board Pass Rate Strategies ............................................................ 20 Physician Assistant and Board Pass Rates ................................................................................ 21 Athletic training and Factors Affecting First-Time Board Exam Pass Rates ........................... 22 Rationale.................................................................................................................................... 29 Method .......................................................................................................................................... 29 Study Design ............................................................................................................................. 29 Participants ................................................................................................................................ 30 Data Collection.......................................................................................................................... 30 Data Analysis ............................................................................................................................ 32 Results ........................................................................................................................................... 33 Discussion ..................................................................................................................................... 35 Limitations and Barriers of Study ............................................................................................. 42 Recommendations for Future Research .................................................................................... 43 Conclusion................................................................................................................................. 44 Appendix A ................................................................................................................................... 60 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 3 List of Tables Table 1 Description of Athletic training Programs ....................................................................... 56 Table 2 Student Demographics by Program ................................................................................. 57 Table 3 Outcomes by those who passed and did not pass the BOC ............................................. 58 Table 4 Logistic Regression Predicting Likelihood of Passing the BOC based on Timing of Distribution, GPA, and ACES Scores........................................................................................... 60 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 4 Abstract Major changes have taken place in athletic training education over the past 15 years. With the changes came an increased expectation for reporting quality program outcomes. One important outcome places emphasis on first-time Board of Certification (BOC) exam pass rates. The purpose of this study was to examine the relationship between academic, pre-admission, and programmatic variables to first-time BOC exam pass rates. This retrospective study sampled 210 students from three Commission on Accreditation of Athletic training Education (CAATE) programs whose students attended an Amato-Cole Educational Services (ACES) Preparatory Workshop before taking the BOC examination. The research suggests that students with higher GPA, ACES Test scores, and distribution of the BOC Practice Analysis during the junior or senior year had higher first-time BOC exam pass rates. The logistic regression model was statistically significant, X2 (3) = 57.92, p < .001. The model explained 39.3% (Nagelkerke R2) of the variance for passing the BOC exam and correctly classified 83.3% of cases. This study revealed the final cumulative GPA, coupled with an ACES Test 2 evaluation, might provide valuable data when preparing students for the BOC exam and ensuring student success. Students with lower cumulative GPA and ACES Test 2 scores may need additional intervention and instruction. Ensuring program quality and student excellence are keys to the athletic training profession maintaining a valuable presence in today's healthcare system. Keywords: Amato-Cole Educational Services, athletic training student, Board of Certification, athletic training education, ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 5 Acknowledgement I would like to express my gratitude to the many people who helped me in this endeavor. Without their support I would not have been able to complete this program and research project. I would first like to thank my research project committee for their commitment to this project. To my committee chair, Dr. Christine Lauber, for your expertise in this area of research and for your continuous support, guidance, and encouragement. To Dr. Elizabeth Moore, whose statistical expertise, suggestions, and valuable feedback were truly the key to bringing this project together. To my friend, Dr. Herb Amato, for your content expertise and many conversations prior to this journey that ultimately led to the proposed area of research. I would like to thank all of the faculty and staff within the doctor of health science program at the University of Indianapolis. This experience has been one I will always cherish and fondly reflect upon. I would be remiss if I didnt acknowledge the awesome support of administration, colleagues, faculty, and athletic training students at Central Methodist University, my second family for the past 27 years. To my parents, who instilled in me that all things are possible through hard work, perseverance, and humility. I dedicate the last three years to my dad who passed away three days before day one of this journey. To my sons Dr. Mackenzie Thiessen and Tanner Thiessen for their unwavering support, humor, and love during this process. Last but not least, I will be forever grateful for the love and encouragement of my wife Sonya for without her this would not have been worth the toil. ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 6 Factors Associated with First-Time Pass Rates on the Athletic Trainers Board of Certification Examination Graduates from Commission on Accreditation of Athletic training Education (CAATE) athletic training programs are expected to be competent in injury and illness prevention; wellness promotion and education; emergent care; examination and clinical diagnosis; therapeutic intervention; and rehabilitation of injuries and medical conditions. To practice as an entry-level credentialed athletic trainer, students must pass the Board of Certification (BOC) examination (BOC, 2017). Each year, over 4,000 students graduate from CAATE programs and take the BOC examination (BOC, 2019). In 2018-2019, the first-time BOC exam pass rate was 73.2% (BOC, 2019). Failure to pass the BOC exam not only affects athletic training program outcomes and student recruitment but also adds a cost burden and delay of employment opportunities for the student (BOC, 2019). Problem Statement In 2012, CAATE implemented a standard (Standard 11) requiring all accredited athletic training programs to maintain a 70% three-year aggregate first-time BOC exam pass rate (CAATE, 2012). Athletic training programs were required to be compliant with Standard 11 by 2013, and professional programs not compliant by 2016 were placed on probation (CAATE, 2016a). In 2016, the 2015-2016 CAATE Analytics Report indicated 91 of 370 programs, over 25%, were non-compliant with the standard and in jeopardy of having accreditation withdrawn (CAATE, 2016b). Since the introduction of Standard 11, athletic training program directors face the task of identifying metrics that can be used to predict students most likely to pass the BOC exam on the first attempt. Many programs have used metrics including: American College Testing (ACT) or ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 7 Scholastic Assessment Test (SAT) scores, athletic training program grade point average (GPA) (Bruce et al., 2016; Harrelson, Gallaspy, & Leaver-Dunn, 1997; Hobson, 2019; Lawrance & Lauber, 2015; Murray, 2014; Platt, Turocy, & McGlumphy, 2001), and other various academic variables to predict success. Program directors have also investigated strategies that could be used by students to improve their chances of passing the BOC exam on the first attempt (National Athletic Trainers Association [NATA], 2013). The Amato-Cole Educational Services (ACES) Preparatory Workshop is used by some athletic training programs to help athletic training students prepare for the BOC exam (H. K. Amato, Email correspondence, April 1, 2019). While many health professions have sought various tests in preparation for their respective board exams, evidence of their predictive value is limited. The profession of athletic training currently does not have a predictive metric for the BOC examination. While many athletic training programs use the ACES Preparatory Workshop to increase BOC pass rates, its effectiveness of improving BOC exam first-time pass rates has not been researched. Study Purpose The purpose of this study was to explore whether participation in an ACES Preparatory Workshop, undergraduate assessments, and institutional factors would have an impact on the first attempt pass rate of students who took the BOC exam. Specifically, this study sought to answer the following research question: For athletic training students who take the BOC exam, what factors are associated with first attempt pass rates? To determine if there was a statistically significant difference between students who passed the BOC exam on the first attempt and those who did not pass it on the first attempt, the following factors were tested: ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 8 1. Institutional factors (athletic training student enrollment, Carnegie classification, when the BOC Practice Analysis was distributed to students, number of core athletic training faculty, Standard 11 compliance, and study guide use). 2. Student characteristics (age, gender, and primary language). 3. Undergraduate assessments (first attempt BOC exam results, ACES Preparatory Workshop Test 2 scores, ACT/SAT scores, and final cumulative GPA). Results from this study may help athletic training program directors and educators identify variables that may predict their students first-time BOC exam pass rate. Insight into what factors forecast student success may drive policy, curriculum, and BOC study strategy changes needed to address potential student and program deficiencies. Literature Review Health professions are faced with examining entry-level degree requirements to secure a place in providing quality healthcare in todays marketplace. Athletic training is one of the allied health professions seeking to be part of the conversation. In 2012, the National Athletic Trainers Association (NATA) Board of Directors accepted the recommendations of a report (Professional Education in Athletic Training: An Examination of the Professional Degree Level) on the future direction of athletic training education presented by the Executive Committee for Education (NATA, 2013). Among the initiatives in this report was the reexamination of the degree level appropriate for the preparation of athletic trainers. In 2013, the NATAs Executive Committee for Education formed a working group to examine the appropriate degree level to enter an athletic training program. The working group presented the findings to the Strategic Alliance (BOC, CAATE, NATA, and NATA Foundation). The Strategic Alliance then recommended to the NATA Board of Directors to change the entry-level degree in athletic training from a ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 9 bachelor's degree to a masters degree (NATA, 2013). In 2015, the NATA Board of Directors voted to accept the recommendation from the Strategic Alliance to move the professional degree in athletic training to the masters level (CAATE, 2015; NATA, 2013). The investigation into the degree change in athletic training revealed several factors linked to successful performance on the BOC examination. These factors included academic aptitude, program quality, length of the program, student age, and completion of a masters degree (NATA, 2013). This information, in part, influenced the recommendation to change the degree level (NATA, 2013). Aggregate data from 2010-2013 athletic training master's degreeprofessional programs indicated program quality and student age as factors of BOC exam success. Master's degree programs demonstrated a 96% compliance with the CAATE first-time BOC exam pass rate standard. Whereas, in athletic training undergraduate programs, only 57% met the required 70% three-year first-time pass rate (NATA, 2013). Since the initial conversation and subsequent inception of the first-time BOC exam pass rate Standard 11 in 2012, there seems to be a concerted focus on the variables related to a successful outcome on the BOC exam. As a result of the concentration of Standard 11, the percentage of undergraduate programs not meeting the 70% pass rate decreased from 43% in 2010-2012 to 28% in 2011-2013 (NATA, 2013). While the BOC first-time pass rate minimum percentage (Standard 11) was implemented before the decision to transition to the master's degree, the first-time BOC pass rate remains a crucial component of assessment to both undergraduate and masters level athletic training programs. As athletic training education programs strive to remain relevant in health care and prepare for a professional degree change, program directors must evaluate admissions criteria and other metrics of success to compete for students and ensure the delivery of quality healthcare from its graduates. ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 10 Driven by the competitive climate of healthcare and the Institute of Medicine's (IOM) initiative to transform the system to provide improved health outcomes, it is essential that allied health programs better prepare the workforce to create a more cost-effective, evidence-based patient-centered care delivery (IOM, 2001). In a climate of rising educational costs and the constant struggle for student recruitment and retention among colleges and universities, comes consumer scrutiny and the demand for successful outcomes. While the Higher Education Opportunity Act of 2008 requires institutions of higher education to publish program and student outcomes (National Postsecondary Education Cooperative, 2009), many health profession education accrediting agencies began to require the publication of available metrics to demonstrate program success. In athletic training, as with other health professions requiring standardized board exams, the most evident measure of student and program success is the firstattempt pass rate of national board exams. The following literature review provides a history of the BOC, CAATE, and the ACES Preparatory Workshop. Moreover, it examines the available research concerning criteria used to ensure success on national board examination pass rates in the professions of athletic training, occupational therapy, physical therapy, nursing, and physician assistant (PA). While the quest for predictive variables affecting board exam outcomes has led to varied results, current literature may give valuable insight into the evaluative process for individual programs seeking to strengthen success rates on board exams and demonstrate their programs educational value. Historical Perspective of Athletic Training Education Athletic training education has gone through many modifications since the formation of the NATA in 1950. Delforge and Behnke (1999) assembled an early timeline on the history and evolution of athletic training education. In the beginning, William E. Newell, ATC, PT. of ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 11 Purdue University was appointed as the National Secretary of the NATA, the position known today as the Executive Director. Perhaps the most significant act of Newells tenure was the formation and appointment of a Committee on Gaining Recognition. The committees charge was to focus on professional development and advancement of athletic training. In 1955, the NATA Board of Directors approved the committee to prioritize focus on the development of a model curriculum and a national certification examination. In response to the Board of Directors vision, NATA Professional Education Committee (PEC) and NATA Certification Committee were formed, with individual attention to education and certification respectively. Delforge and Behnke (1999) also chronicled the development of the athletic training certification examination. In December 1968, the NATA Board of Certification (NATABOC) was formed. While the NATA PEC was developing educational programming throughout the 1950s and 1960s, the first national certification examination was simultaneously formed by the NATA Certification Committee. Before the administration of the first certification exam in August of 1970, it was declared that all active members of the NATA were eligible for automatic certification through a grandfather clause. However, the grandfather clause closed after December 31, 1969 (BOC, 2007; Delforge & Behnke, 1999). There were five routes to certification: 1) graduate from an approved athletic training curriculum program; or 2) graduate from a 4-year college or university and proof of five years of experience; or 3) obtain a physical therapy degree with two years of athletic training experience; or 4) complete 1800 hours of on-the-job apprenticeship training; or 5) qualify for special consideration including state teaching licensure and NATA- approved workshop completion (BOC, 2007; Delforge & Behnke, 1999) ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 12 Grace (1999) further established a timeline of athletic training certification. The first NATABOC exam consisted of two sections. Section One contained questions of basic and clinical science, and Section Two focused on theory and application of athletic training knowledge and skills. In 1978, the National Commission for Health Certifying Agencies (NCHCA) was established by the U.S. Department of Health, Education, and Welfare. The purpose of this agency was to provide standards that healthcare education programs would follow to ensure the proper development of healthcare academic programs. In addition, the agency also ensured that certification credentials awarded by those programs were vetted using widely accepted testing, psychometric, and legal principles. Grace (1999) noted other milestones in athletic training credentialing. In 1983, the NATA was granted NCHCA approval and became its own credentialing body. In 1987, the NATA registered ATC as the designation of a fullycredentialed athletic trainer through the U.S. Patent Office. In 1989, the NATABOC became a separate organization with all rights to certify practitioners. It later changed its name to the Board of Certification (Delforge & Behnke, 1999). In June 1990, the American Medical Association (AMA) formally recognized the profession of athletic training as an allied health profession (Delforge & Behnke, 1999). Once athletic training was recognized as an allied health profession, the process of accrediting programs began. Initially, the AMA's Committee on Allied Health Education and Accreditation (CAHEA) began developing requirements for entry-level athletic training programs. During this time, programs voluntarily seeking accreditation were required to follow the CAHEA accreditation process (Delforge & Behnke, 1999). The Commission on Accreditation of Allied Health Education Programs (CAAHEP) later replaced CAHEA and directed the accreditation process. The Joint Review Committee on Educational Programs in Athletic training (JRC-AT) ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 13 was incorporated in 1991 as a committee of CAAHEP (Delforge & Behnke, 1999). In 2006 the JRC-AT became an independent accrediting agency and officially changed its name to the Committee for Accreditation of Athletic training Education (CAATE, 2014). Commission on Accreditation of Athletic training Education is currently responsible for the accreditation of more than 390 education programs and collaboratively continues to develop professional standards for entry-level athletic training programs (CAATE, 2019a). From 1970 through 2004, applicants wishing to take the BOC exam were required to either graduate from a NATA-approved curriculum program or graduate from an accredited college or university and complete internship requirements (BOC, 2007). Since 2004, athletic training certification can only be achieved by graduating from an athletic training program accredited by CAATE. The current electronic format of the BOC exam was introduced in June 2007 (BOC, 2007). Historical Perspective of Amato-Cole Educational Services Preparatory Workshop Board of Certification exam scores for athletic training programs in the early 1990s revealed low national first-time BOC pass rates. In 1992, Herb Amato, then program director at James Madison University, was developing ways for students in the universitys athletic training program to be more successful on the BOC exam (H. K. Amato, Email correspondence, April 1, 2019). The result was a series of review sessions for students. (H. K. Amato, Email correspondence, April 1, 2019). H. K. Amato (Email correspondence, April 1, 2019) indicated the weekly review series was fostered over three years and aimed at providing students with a series of tests and study materials to identify areas of focus within the domains of the BOC Role Delineation Study (now called the BOC Practice Analysis) The first mock exam was created in ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 14 the fall of 1991 from an assignment in a doctoral course at Middle Tennessee State University (H. K. Amato, Email correspondence, April 1, 2019). In the spring of 1995, Steve Cole, the athletic training program director, and head athletic trainer at the College of William and Mary (W&M) asked Dr. Amato to prepare his athletic training internship students for the upcoming BOC exam (H. K. Amato, Email correspondence, April 1, 2019). The weekly review sessions were combined and taught in a weekend course for W&M seniors. After this event, Dr. Amato and Mr. Cole decided the review session for the W&M students should be expanded and offered to others (H. K. Amato, Email correspondence, April 1, 2019). H. K. Amato (Email correspondence, April 1, 2019) noted that Steve Cole suggested the workshop be presented to other athletic training programs and athletic training students, and Amato-Cole Educational Services (ACES) was born. The goals of ACES are to identify areas of deficiency, develop a study plan of action based on identified deficiencies, reduce exam fears and test day anxiety, and expose students to questions similar to those on the BOC exam (H. K. Amato, Email correspondence, April 1, 2019). In 2008, ACES signed a contract with the NATA to administer two of the five available ACES exams on-line. The additional two ACES exams are given during the ground-based workshop, and a fifth exam is used in whatever manner a student or AT program director believes to be the most appropriate for future success (H. K. Amato, Email correspondence, April 1, 2019). Nursing and Predicting First-Time Board Exam Pass Rate Due to a potential nursing shortage and an increase in projected employment opportunities exceeding 3.3 million by 2026 (Unites States Department of Labor, 2018), nursing programs are faced with the task of increasing program enrollment, increasing nursing program student retention rates, and preparing more nursing school graduates (Herrera & Blair, 2015). ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 15 Research in the areas of nursing program retention and student preparation strategies has indicated minimum GPA and prerequisite course curriculum requirements as a plausible way to increase nursing school enrollment and identify students more likely to stay in nursing programs (Crouch, 2015; Herrera & Blair, 2015). Researchers also found the use of standardized tests could not only predict students more likely to stay in the program resulting in increased retention, but they also could be used to increase pass rates on the NCLEX-RN examination (Crouch, 2015; Herrera & Blair, 2015). Standardized tests found to be useful in nursing are the Health Education System Incorporated (HESI) admission assessment (A2) test which helps predict academic readiness for nursing program admittance, and the HESI exit exam (E2) to examine academic preparation and predict outcomes on the National Council Licensure Examination for Registered Nurses (NCLEX-RN) (Sosa & Sethares, 2015). While admission, subject matter, and mid-curricular exams exist for HESI, the focus appears to be on the ability of the E2 to predict success on the NCLEX-RN (Sosa & Sethares, 2015). A HESI validation study found the E2 to not only be a reliable predictor of student success on the NCLEX-RN but also provide objective data when evaluating progress within the nursing curriculum when coupled with HESI specialty exams that contain critical thinking and problem-solving test questions (Zweighaft, 2013). Five component scores of the A2: basic math, reading comprehension, grammar, vocabulary and general knowledge, and anatomy and physiology were investigated to determine whether the A2 affected successful completion of three associate degree nursing courses (Chen &Voyles, 2013). Results indicated a positive correlation between all A2 scores and nursing students successfully completing all three nursing courses. This study indicated that A2 scores could be used to predict student success as determined by successful course completion in the first semester of a nursing curriculum (Chen ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 16 &Voyles, 2013). The same study found that A2 scores can aid in student retention, which could result in lowering the cost associated with attrition (Chen &Voyles, 2013). Other studies have compared the relationship of the HESI A2, preadmission GPA, science GPA, and timely progression to passing the NCLEX-RN exam. Findings suggested preadmission GPA and science GPA had good predictability of timely program progression; however, the A2 did not correlate with preadmission GPA or science GPA. Researchers reported that it was imprudent to relate success on the A2 with passing the NCLEX-RN exam for registered nurses (Hinderer, Dibartolo, & Walsh, 2014). While nursing programs with high application rates and low seat availability are striving to see students successfully matriculate through the program, Hinder et al. (2014) suggested caution when considering preadmission GPA, science GPA, and the A2 as indicators of success on the NCLEX-RN. Nursing board councils occasionally field claims that the NCLEX exam is biased against candidates who list English as their second language (ONeil, Marks, & Liu, 2006). ONeil et al. (2006) investigated the impact of English as a second language on nursing students passing the NCLEX-RN and National Council Licensure Examination for Practical Nurses (NCLEX-PN) on their first attempt. The researchers found nursing students who listed English as a second language or that English and another language as their primary language had 10%- 15% lower pass rates compared to those students who listed English as their primary language (ONeil et al., 2006). Predictors of First-Time Pass Rate on the National Physical Therapy Examination In physical therapy, an increase in student applications and competition for students by physical therapy programs has increased the importance of predicting success on the National Physical Therapy Examination (NPTE) (Utzman, Riddle, & Jewell, 2007). Several researchers ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 17 have examined various factors that may contribute to success or failure on the NPTE. Utzman et al. (2007) examined pre-admission variables that had the potential to predict academic difficulties within physical therapy education programs. Variables included demographics (age and race or ethnicity) and pre-admissions data (undergraduate GPA, quantitative and verbal GRE [Graduate Record Exam] scores) (Utzman et al., 2007). Any student having to repeat a course or being placed on academic probation, suspension, or dismissed from the education program was considered as having academic difficulty (Utzman et al., 2007). The researchers concluded that the use of undergraduate GPA, GRE scores, as well as age and race or ethnicity, were useful in predicting student academic difficulty in physical therapy education programs (Utzman et al., 2007). Building on research correlating the GRE scores and undergraduate GPA to physical therapy program success (NPTE pass rate), a retrospective study was designed to investigate whether the Health Science Reasoning Test (HSRT) along with GRE scores and undergraduate GPA were related to first-time pass rate on the NPTE (Huhn & Parrot, 2017). The HSRT is a standardized test measuring clinical reasoning skills for students in healthcare professions. Researched as a part of an admissions decision tool, the authors recognized the validity of the HSRT in tracking changes in critical thinking (Huhn & Parrot, 2017). Huhn and Parrot (2017) also found the HRST added significantly to the performance of the predictive model when considering undergraduate GPA, final GPA, and GRE scores A 2001 study found a link between physical therapy pre-admission factors and academic success with first-time pass rates on the NPTE (Dockter, 2001). Dockter (2001) found that the best predictor of academic success was the combination of total admission points on admission score criteria and age upon program admission. The best predictor of success on the NPTE was GPA following completion of the first year of physical therapy school (Dockter, 2001). In a ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 18 similar study, admission criteria consisting of GRE scores, overall and pre-requisite preadmission GPA, and admissions committee evaluations were explored as predictors for acceptance into a physical therapy entry-level masters degree program (Thieman, Weddle, & Moore, 2003). The study examined the ability of admissions criteria to predict the final master's program GPA, applied skills using a physical therapy Clinical Performance Instrument (CPI), and NPTE scores (Thieman et al., 2003). Pre-admission GPA, when evaluated with age and GRE scores, accounted for 37% of the variability in prediction of the masters program final GPA (Thieman et al., 2003). The master's degree GPA was the best predictor of NPTE scores. However, when paired with previous grades and GRE scores, only 11% of the variability in NPTE scores was accounted for by the model (Thieman et al., 2003). In addition, the admissions variables did not predict CPI scores (Thieman et al., 2003). A study by Riddle, Utzman, Jewell, Pearson, and Kong (2009) approached academic difficulty and program-level variables to determine performance on the NPTE. The results of this study suggested a significant relationship between academic difficulty and institutional and Carnegie classification as predictors of failure on the NPTE (Riddle et al., 2009). Failure on the NPTE was dependent on the schools Carnegie classification further noting that private masters degree programs had 2.78 times higher odds of NPTE failure than for students in a public masters degree program (Riddle et al., 2009). Awareness of academic difficulty and remediation factors may provide programs with an opportunity to develop an intervention strategy or curricular enhancement when concerned with retention and NPTE success (Riddle et al., 2009). In other research, Crawley, Greene, Brown-White, Simpson, and Karavatas (2015) surveyed study habits of physical therapy students to see if there was an effect on first-time pass rates for the NPTE. The authors found no significant relationship to first-time NPTE pass rate when ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 19 taking into consideration study materials, the number of hours a day spent studying, the number of days a week spent studying, and the number and type of practice tests taken (Crawley et al., 2015). Foundational courses, program GPA, and verbal, quantitative GRE scores were found to be the best predictors of first-time pass rates on the NPTE when including variables of the Clinical Performance Evaluation Instrument and academic difficulties (Bayliss, Thomas, & Eifert-Mangine, 2017). Another study investigated the variables of age at graduation, professional-grade point average, overall exam score, and scores on a physical therapy clinical performance instrument (Kosmahl, 2005). Many allied health care education programs focus on various instruments to monitor a students clinical progression and employ a comprehensive exam to track academic retention in an attempt to ensure professional competence and success on their respective board exams (Kosmahl, 2005). Similar to other studies (Dockter, 2001; Huhn & Parrot, 2017; Thieman et al., 2003), this study revealed a significant positive correlation with professional GPA and scores on the NPTE. However, there was no correlation between age and clinical performance scores on NPTE test scores (Kosmahl, 2005). Comprehensive exams and research in curriculums have been compared to first-time board pass rates with mixed conclusions. Kosmahl (2005) found a significant correlation between comprehensive exams and scores on the NPTE and reinforced the notion of closely monitoring a students academic performance as well as the use of a comprehensive exam to identify weaknesses in curricular content prior to taking the NPTE. A study comparing the volume of published research artifacts and pass rates on the NPTE revealed no significant association between the number of artifacts submitted and NPTE pass scores concluding that ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 20 other factors are likely better at predicting pass rates for the NPTE (Cook, Landry, Covington, McCallum, & Engelhard, 2015). Clinical education is a crucial component of many healthcare education programs. Luedtke-Hoffmann, Dillon, Utsey, and Tomaka (2012) examined scores from the Physical Therapist Manual for the Assessment of Clinical skills (MACS), final physical therapy program GPA, compared to NPTE scores for first-time test takers. While results of this study did find a correlation between two skill sections of the MACS and NPTE exam, the coefficients were too low to infer any meaningful correlation. Luedtke-Hoffmann et al. (2012) did find a positive correlation between GPA and NPTE scores. This study reiterates the use of GPA as an indicator of standardized test scores and suggests using clinical assessment tools to ascertain clinical weaknesses for professional skill development (Luedtke-Hoffmann et al., 2012). Occupational Therapy and Board Pass Rate Strategies The Accreditation Council for Occupational Therapy Education (ACOTE) requires programs to possess a 3-year average pass rate of 80% or higher adding to the importance of examining variables which may predict success on the National Board for Certification in Occupational Therapy (NCBOT) (Novalis, Cyranowski, & Dolhi, 2017). Novalis et al. (2017) suggested that variables of gender, recommendation strength scores, program GPA, and fieldwork self-evaluation scores significantly predicted outcomes on the NBCOT examination. This model correctly predicted 82.9% of student success on the NBCOT exam (Novalis et al., 2017). An online pilot course, with its primary purpose of successfully preparing occupational therapy students to complete the NBCOT examination was developed (Breen-Franklin, 2017). This pilot program provided materials, assignments, study plans, and practice tests for students ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 21 who were preparing for the NBCOT exam. A review of the pilot course and a two-day commercial course revealed high pass rates among both programs suggesting the use of similar programs when designing strategies to prepare for board examinations successfully (BreenFranklin, 2017). MacArther and Randall (2015) investigated the extent in which GRE scores, undergraduate GPA, prerequisite course grades, applicant essay scores, and occupational therapy school grades might predict occupational therapy fieldwork scores and first-time pass rates on the NBCOT exam. While regression models in this study provided little insight into variables that could increase the predictability of fieldwork scores, models were better at predicting NBCOT first time pass rates. The model indicated increased predictability when using GRE scores and occupational therapy school grades (MacArther & Randall, 2015). While all courses within the program were evaluated, only three (Gerontology, Biomechanical Treatment, and Adult Neurological Treatment) were found to predict first-time pass rates on the NBCOT exam. The inclusion of preadmission variables in the model as significant indicators of first-time NBCOT pass rates underscores the need for programs to evaluate preadmission criteria and program GPA when considering program admittance and types of remediation for board preparation (MacArther & Randall, 2015). Physician Assistant and Board Pass Rates Researchers have investigated various factors that may predict success on the Physician Assistant National Certifying Exam (PANCE) including GRE scores, program selection criteria, and other forms of measured student success. Hocking and Piepenbrock (2010) reviewed websites of physician assistant programs and found that 55% utilized the GRE as part of the admissions process. While the GRE is the most widely used standardized admissions test for ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 22 masters level PA programs, many (54%) do not require a minimum score, and only 40% of those programs use it according to published GRE guidelines (Hocking & Piepenbrock, 2010). It may be challenging to determine the usefulness of the GRE in an admissions process (Hocking & Piepenbrock, 2010). Other researchers examined and compared first-time pass rates on the PANCE with student success (PANCE pass/fail and score, and program GPA) and pre-admission criteria (undergraduate GPA, prerequisite course institution type, and prior healthcare experience) (Brown, Imel, Nelson, Hale, & Jansen, 2013). Brown et al. (2013) found no correlation between student success and pre-admission criteria. However, there was a strong or moderate association between pharmacology and anatomy grades and program GPA but not with PANCE pass rates. (Brown et al., 2013). Athletic training and Factors Affecting First-Time Board Exam Pass Rates Brown, Potteiger, Keeley, Lundgren, and David (2017) reviewed professional undergraduate athletic training programs (ATPs) and found the most common admissions decision process came in the form of a point system, criterion list, or committee perception among other methods of student selection. In this review, major GPA, overall GPA, or a combination of the two, as well as prerequisite courses, clinical hours, and personal attributes through the use of interview, resumes, letters of recommendation or other written forms of application materials were commonly used in the selection process (Brown et al., 2017) Many athletic training programs have used admission application requirements as a method of ensuring program and ultimately national board exam success. Authors investigated minimum program GPA, grades on prerequisite coursework, student evaluations and documented direct observations, application essay, admittance exam, an interview process, and letters of recommendation as part of an entry-level masters athletic training program application ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 23 process. (Hawkins, McLoda, & Stanek, 2015). The authors found a decrease in program attrition, and an increase in students who took the BOC examination when using the multifaceted approach to admitting students into the masters program (Hawkins et al., 2015). The researchers also investigated data from nursing and physical therapy literature and found similar results and benefits of using a combined approach which included GPA, application essays, observation evaluations, prerequisite coursework grades, and interviews when considering criteria for program admission (Hawkins et al., 2015). Researchers also examined preadmission criteria as predictors of academic success determined by college GPA. Platt, Turocy, and McGlumphy (2001) investigated graduates from six allied health education programs from one private university. Programs included athletic training, occupational therapy, health management system, perfusion technology, physician assistant, and physical therapy (Platt et al., 2001). Platt and colleagues (2001) investigated student admission variables of high school GPA, SAT verbal scores, SAT mathematics scores, and SAT total scores, compared to college GPA. Results demonstrated that high school GPA and the verbal portion of the SAT were predictive of college GPA when compared to the allied health programs as a whole (Platt et al., 2001). However, this only accounted for a small portion of the variance of success when comparing college GPA (Platt et al., 2001). The athletic training program in the study found that high school GPA predicted more than a third of the program's success in college GPA (Platt et al., 2001). Harrelson et al. (1997) found a composite set of variables consisting of overall GPA, athletic training GPA, minor academic GPA, ACT composite score, and the number of semesters at the university predicted first-time BOC exam success. However, no single variable predicted first-time BOC exam pass rates (Harrelson et al., ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 24 1997). Three of the variables were related to GPA, which may indicate the importance of GPA in predicting BOC exam pass rates (Harrelson et al., 1997). Keskula, Sammarone, and Perrin (1995) examined five predictive variables used in the selection of students into a NATA-approved graduate athletic training education program. Among those variables were the quantitative scores and verbal scores of the GRE, preadmission GPA, preadmission clinical hours, and undergraduate education (curriculum or internship) as compared to the final graduate program GPA (Keskula et al., 1995). Contrary to the SAT findings by Platt et al. (2001), the verbal portion of the GRE was not found to have a significant effect on the final program GPA. Similarly, preadmission GPA was a significant factor in the prediction of the final program GPA (Platt et al., 2001). A study by Lawrance and Lauber (2015) compared high school academic performance (GPA, standardized test scores, number of college preparatory courses taken, and quality points) and professional athletic training program academic performance (undergraduate GPA, athletic training major GPA, end of sophomore year GPA, and select course GPA) with first-time pass rates on the BOC exam. The authors found that students who had a higher high school GPA and standardized test scores were more successful in the professional athletic training program (completed program without probationary status) (Lawrance & Lauber, 2015). Lawrance and Lauber (2015) found standardized test quantitative scores proved to be the best predictor of students first-time BOC exam pass rates. While athletic training program coursework was a significant variable predicting BOC pass rates, it was only a small portion of the variance found in this study (Lawrance & Lauber, 2015) Studies have sought to identify a predictive model for success in a professional masters degree in athletic training. One study identified three pre-admission criteria as predictors of success in a professional masters athletic training program where success was defined as ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 25 graduate program GPA (gGPA) after completing the first year of the curriculum (Bruce et al., 2016). The three-factor model for prediction of success was identified as undergraduate GPA (uGPA) 3.18; GRE scores 145.5, and completing calculus as an undergraduate student. The pre-admission criteria model found that students who had any two of the pre-admission predictors were 2.75 times more likely to be successful in the graduate program (Bruce et al., 2016). Additionally, Bruce et al. (2016) found a student having a gGPA of 3.45 at the end of the first year of graduate study was successful within the graduate program. Using the 3.45 gGPA as a cut point, it was determined that students who had a gGPA 3.45 were 8.3 times more likely to pass the BOC exam on the first attempt compared to those who had a gGPA 3.45 at the end of the first year of athletic training graduate studies (Bruce et al., 2016). This study suggests it is possible to create a model that may be beneficial for programs to predict student program success when measured by graduate program GPA and success on the BOC exam first-time pass rate (Bruce et al., 2016). Completing clinical experiences are an essential component of health care education programs, and athletic training is no different. One study sought to determine if there was an association of student sex, overall program GPA, and the number of clinical hours completed with performance on the BOC examination (Middlemas, Manning, Gazzillo, & Young, 2001). During the time of this study, the BOC exam was a three-part exam consisting of written, practical, and simulation sections. Analysis indicated no significant difference regarding the sex of the participant and passing all three parts of the BOC exam (Middlemas et al., 2001). There was no significant correlation between the number of clinical hours and scores on all three portions of the BOC (Middlemas et al., 2001). However, when examining GPA, there was a significant association with scores on all portions of the BOC exam (Middlemas et al., 2001). A ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 26 similar study compared student age, sex, the route to certification, previous athletic training, and allied health experience, and clinical education experiences to scores on the BOC exam (Turocy, Comfort, Perrin, & Gieck, 2000). Age was found to have an association with the oral practical scores of the BOC exam (Turocy et al., 2000). Older candidates were more successful than younger candidates on the oral practical portion of the BOC exam, but it did not affect the written or written simulation scores of the BOC exam (Turocy et al., 2000). Turocy et al. (2000) reported that students who gained more clinical hours above the minimum hour requirement gained no more significant benefit in passing the BOC exam on the first-attempt than students who acquired fewer hours. While the clinical component of athletic training education is vital for professional practice, it would appear other variables may be more predictive of success on the BOC exam. More recently, there has been increased attention to factors that may attribute to first-time BOC exam pass rates for students in professional masters degree athletic training programs. In one study, 11 athletic training programs were surveyed to obtain various program requirements, cohort characteristics, and outcome measures, including BOC exam pass rates. (Ostrowski & Marshall, 2015). The study considered the association between programs with structured research requirements to first-time BOC exam pass rates, and between programs that required students to take a program-specific comprehensive examination to first-time BOC exam pass rates (Ostrowski & Marshall, 2015). There was no correlation with programs that required a comprehensive exam and first-time pass rates on the BOC exam (Ostrowski & Marshall, 2015). However, there was a significant difference between programs that had a structured research component and first-time BOC pass rates (Ostrowski & Marshall, 2015). Programs that had a structured research component in the curriculum demonstrated significantly higher first-time ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 27 BOC exam pass rates than those programs that did not have a structured research component in the curriculum (Ostrowski & Marshall, 2015). One program that had no programmatic research requirements had the lowest BOC exam pass rate among programs studied (Ostrowski & Marshall, 2015). Another study investigated whether there was a relationship among admissions GPA, final GPA, gender, and age to first-attempt BOC pass rates (Murray, 2014). A relationship was determined to exist between student admission GPA and final program GPA with firstattempt BOC exam pass rates (Murray, 2014). Students face many challenges while preparing for the BOC exam. A study by Breitbach, Downey, and Frager (2013) focused on students academic worries, locus of control (perception of ability to control one's fate), and coping style (emotional focus versus problem-focused). The researchers hypothesized that coping style would negatively affect testing results if the student's coping mechanism focused on managing emotions associated with the stress of test preparation (emotion-focused) rather than a more step-by-step approach to changing the situation or altering the cause of the testing stress or anxiety (problem-focused) (Breitbach et al., 2013). Using a more problem-focused strategy to cope with test stress and anxiety when preparing for the BOC exam may help students counteract negative emotions tied to the testing experience (Breitbach et al., 2013). The researchers suggested that students who had high academic worry and utilized emotional focused coping skills did not do well on the BOC exam on the first attempt (Breitbach et al., 2013). Students who employed a problem-focused coping strategy were more successful in passing the BOC exam on the first attempt (Breitbach et al., 2013). One of the goals of the ACES Preparatory Workshop is to reduce exam and test day anxiety by offering students a step-by-step (problem-focused) blueprint to overcoming test anxiety (H. K. Amato, Email correspondence, April 1, 2019). ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 28 Lastly, a recent study scrutinized the accuracy of pre-professional athletic training program admission criteria and professional athletic training program metrics as predictors of first-time BOC exam pass rates (Hobson, 2019). Measures used in program admissions criteria were pre-professional phase program course GPA, cumulative GPA upon admission, average admissions interview evaluation scores, average scores from preceptor evaluations, and use of a single composite score generated inclusively of all previously mentioned pre-program admissions data (Hobson, 2019). The single composite score was weighted and given a value ranging from 0 to 10, and it was investigated to provide a comprehensive guide regarding the admissions process (Hobson, 2019). Additionally, the researchers compared variables within the professional athletic training program (final professional program GPA, combined math and reading SAT scores and clinical education hours) and first-time BOC exam pass rates (Hobson, 2019). Hobson (2019) found the most accurate predictor of first-time BOC exam pass rates within the pre-professional program metrics was the pre-professional phase course GPA. However, he also found that the highest accuracy rating of all examined variables was the combined math and reading SAT scores. The single composite score did show fair accuracy in predicting first-time BOC exam pass rates (Hobson, 2019). However, the accuracy of the single composite score was attributed to the GPA outcomes and suggested programs consider academic variables over subjective evaluations (interview and preceptor evaluations) when creating a single composite scoring metric (Hobson, 2019). Per the study authors, results of this study suggested that programs should consider math and reading SAT scores as well as preprofessional course GPA in the application process (Hobson, 2019). A study by Platt et al. (2001) revealed the verbal (reading) portion of the SAT was predictive of professional program GPA while other studies concluded that professional program GPA had a positive effect on first- ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 29 time BOC exam pass rates (Harrelson et al., 1997; Middlemas et al., 2001; Murray, 2014). The use of the reading portion of the SAT may deserve further investigation when admitting athletic training students or developing curricular changes. Rationale In athletic training, as well as other health professions with standardized board exams, the most evident measure of program success is the initial pass rate of national certifying board exams. Healthcare education programs have investigated many factors related to first-time board pass rates. These factors have ranged from pre-admissions criteria and pre-requisite coursework to the student's final program GPA and board review practice exams. Carnegie classification and student study habits have also been studied. While many variables have shown success in retention efforts and academic curricular success, data establishing definitive predictive criteria for first-time board exam pass rates has proven difficult. Only one study in nursing showed that a validated exit exam could be a reliable predictor of student success for the NCLEX-RN examination (Sosa & Sethares, 2015). Athletic training currently does not have a predictive metric for the BOC examination, and the relationship of the ACES Preparatory Workshop Test 2 examination has not been studied. Results from the current study may help athletic training program directors and educators identify variables that may play a role in predicting first-time BOC pass rates. Insight into what factors contribute to student success may drive policy, curriculum, and BOC exam study strategies needed to address student and program deficiencies. Method Study Design This was a non-experimental study employing a retrospective cohort design to determine if first-time BOC exam pass rates of athletic training students could be predicted using ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 30 undergraduate assessments, institutional factors, and student characteristics. The study occurred between January and June 2019. Before data collection, the study was approved by the University of Indianapolis Human Research Protection Program, with a reliance agreement being established between them and the Central Methodist University Institutional Review Board. Participants A convenience sample of athletic training students admitted to one of six university CAATE professional undergraduate programs were included in the study. Program selection was based on differing Carnegie classifications and CAATE Standard 11 compliance (probation or non-probation status). Students from the selected programs must have participated in an ACES Preparatory Workshop and completed the athletic training BOC examination between 2014 and 2019. Students who were not candidates for the BOC examination and did not graduate were excluded. To estimate the minimum sample size need to detect a statistically significant difference between students who passed and did not pass the BOC on the first attempt, an a priori sample size estimation was conducted using G*Power, version 3.1 (Faul, Erdfelder, Lang, & Buchner, 2009). The calculation was based on comparing means between two independent groups and the following parameters, two-tailed test, alpha of .05, power of .80, a moderate effect size of 0.50, and an allocation ratio of .20. From the calculation, it was estimated that a minimum of 228 participants was needed to power the study appropriately. Data Collection Program directors from six CAATE accredited professional ATPs who serve as host sites for the ACES Preparatory Workshop were contacted via phone and email correspondence regarding participation in the study. The primary researcher (W. W.) is the program director for ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 31 one of the participating undergraduate athletic training programs. Program directors were sent study information and instructions and when appropriate, an encrypted email containing a structured self-reported web-administered Excel spreadsheet to log all data (see Appendix A for Excel data spreadsheet). The following institutional factors were collected from each participating program: athletic training student enrollment, Carnegie classification, when the BOC Practice Analysis was distributed to students, the number of core athletic training faculty, and the use of a BOC study guide. The following data were collected from each program for students who completed the BOC exam: demographics (age, gender, and primary language), first attempt BOC exam results - pass or fail, ACES Preparatory Workshop Test 2 score, highest recorded ACT or SAT score, and final cumulative program GPA. For this study, since both ACT and SAT scores were utilized, SAT scores were converted to an ACT equivalent score using an SAT to ACT Concordance Table (ACT, 2019). The ACES test scores were taken from the second test given during the ACES Preparatory Workshop, and raw scores were used. Program directors agreeing to participate were sent a blank Excel spreadsheet with data headings to complete and return to the primary investigator. Program directors were sent a follow-up email within one week of receiving the spreadsheet to see if there were any questions concerning data collection expectations. A second email was sent two weeks after the first reminder email. Program directors who provided data were then contacted by phone to answer any questions or discuss concerns. As data were received from program directors, the primary researcher combined the deidentified data into one Excel spreadsheet. While the primary researcher had access to participant identifiable information at his site during data collection; no identifiers were recorded on the Excel spreadsheet. All data collected were stored on an external hard drive and secured in the ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 32 primary researchers office. Files were backed-up using the University's mainframe system where the primary researcher was employed, the primary researchers encrypted thumb and external hard drive, and University One Drive cloud storage system. A waiver of informed consent was obtained based on the study collecting retrospective de-identified data. Data Analysis Statistical analysis was conducted using the IBM SPSS Statistics for Windows, Version 24.0 (IBM Corp., Armonk, NY). All tests were two-tailed, and an alpha level of less than .05 was considered statistically significant. The Shapiro-Wilk test was used to determine if data were normally distributed and the Levene's test was used to assess equality of variance for interval and ratio data that were found to be normally distributed. Descriptive statistics were used to describe institutional program factors, student characteristics, and student undergraduate assessments. Nominal data are presented as frequencies and percentages, and interval and ratio data as means and standard deviations for data that were normally distributed or medians and interquartile ranges for data that were not normally distributed. Program institutional factors (student enrollment, Carnegie classification, distribution of the BOC Practice Analysis, number of core athletic training faculty, Standard 11 noncompliance, and study guide use) were compared by programs using Fishers exact tests and Mann-Whitney U tests. To determine if there were differences in student characteristics (age, gender, primary language) and undergraduate assessments (ACES Preparatory Workshop Test 2 scores, ACT/SAT scores, and final cumulative GPA) comparisons by participating programs, and by the two BOC groups (passers and non-passers of first-attempt BOC exam) were conducted using Fishers exact test for nominal data and either an independent t-test or MannWhitney U test, as appropriate for interval and ratio data. To compare BOC groups by the ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 33 distribution of the BOC Practice Analysis, a Fishers exact test with post hoc tests were conducted using the Bonferroni correction to reduce the chance of a type I error. Effect sizes were calculated using Cohen's d values for the significant variables in the bivariate comparisons and were interpreted based on the word by Cohen (1988). Binomial logistic regression was used to identify significant predictors of passing the BOC examination on the first attempt. Predictor variables that demonstrated a significant difference between passers and non-passers on bivariate analyses were entered into the model simultaneously using the Enter method. Before and after conducting a logistical regression analysis, the following assumptions were tested using procedures outlined by Field (2017): linearity between continuous independent variables and the logit transformation of the dependent variable; no multicollinearity; and no significant outliers. Results A total of six program directors from CAATE accredited entry-level undergraduate ATPs were contacted regarding study participation. All six programs agreed to provide information. However, only three (50%) programs participated and provided data on 210 students. The description of institutional factors by program can be found in Table 1. Student characteristics by program are presented in Table 2. As can be seen in Table 2, there was a significant difference in first-attempt BOC exam pass rates by program with Program 2 having the highest pass rate. Because there were only four (2.0%) students in which English was not their primary language, their data were not compared by program. All four of the students in which English was not their primary language were from Program 1. Outcomes were prepared for those who passed and failed the BOC on the first attempt. It was found that those programs that used a non-BOC study guide had a higher first-attempt BOC ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 34 exam failure rate percentage than those who used a BOC approved study guide. When comparing program student enrollment, Carnegie classification, and distribution of the BOC Practice Analysis, it was found that the timing of the BOC Practice Analysis was a stronger predictor of BOC success in the study model; therefore, only distribution of the BOC Practice Analysis was chosen for this study. Post-hoc tests were calculated for the BOC Practice Analysis and found students who had the BOC Practice Analysis distributed in the junior or senior year were over three times more likely to pass the BOC exam on the first attempt than students who received the BOC Practice Analysis in the sophomore year. Further outcomes are presented in Table 3 and indicate a statistically significant difference in ACES Test 2 scores, final cumulative GPA, and timing of BOC Practice Analysis between the two BOC groups. Those who passed the BOC exam on the first attempt had higher ACES Test 2 scores and final cumulative GPA. Cohen's effect size was very large (d = 1.20) for ACES Test 2 scores compared to first-time BOC pass rates providing further indication of ACES Test 2 scores providing a meaningful clinical measure. The median and interquartile ranges for final cumulative GPA scores for students who passed the BOC exam on the first attempt was 3.74 (.367) and 3.42 (.477) for students who failed the BOC exam on the first attempt. The groups differed significantly at a .05 alpha (p <.001) and had a medium effect size (d = 0.69), both indicating a reasonable significance in this study. Logistic Regression Analysis Binomial logistic regression was performed to ascertain which institutional factors, student characteristics, and undergraduate student assessments could significantly predict students passing or failing the BOC on their first attempt. Results of bivariate analyses were used to identify potential predictors of the BOC exam pass rate with those in which there was a ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 35 statistically significant difference being entered into the model. The following seven variables were statistically significant and were entered simultaneously into the model: number of core faculty, Practice Analysis, use of a non-BOC study guide, ACT scores, final cumulative GPA, and ACES Test 2 scores. Linearity of the continuous variables concerning the logit of the dependent variable was assessed via the Box-Tidwell (1962) procedure. A Bonferroni correction was applied using all seven terms in the model resulting in a statistical significance being accepted when p < .010 (Tabachnick & Fidell, 2014). Based on this assessment, all continuous independent variables were found to be linearly related to the logit of the dependent variable. There were no studentized residuals greater than + 3 standard deviations indicating no outliers were unduly affecting the results. The logistic regression model was statistically significant, X2 = (3) 57.92, p < .001. The model explained 39.3% (Nagelkerke R2) of the variance in passing the BOC exam on the first attempt and correctly classified 83.3% of cases. The sensitivity was 40.0%, specificity was 93.9%, positive predictive value was 61.5%, and the negative predictive value was 86.52%. Of the seven predictor variables, three were statistically significant in the final model, final cumulative GPA, ACES Test 2 score, and timing of distributing the BOC Practice Analysis (as shown in Table 4). Higher GPA, ACES Test 2 scores, and distribution of the BOC Practice Analysis in the junior or senior year were associated with an increased likelihood of passing the BOC exam on the first attempt. Discussion The purpose of this study was to explore whether participation in an ACES Preparatory Workshop, institutional factors, student characteristics, and undergraduate student assessments would have an impact on the first-time pass rate of students who take the BOC exam. Results of the data analysis indicated a significant relationship between the final cumulative GPA and first- ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 36 time BOC exam pass rate, the timing of distributing the BOC Practice Analysis and first-time BOC exam pass rate, and a significant relationship between ACES Test 2 scores and first-time BOC exam pass rate. Other variables (student enrollment, Carnegie classification, number of core athletic training faculty, BOC study guide use, non- BOC study guide use, student age, gender, primary language, and ACT/SAT scores) were not significant factors for first-time BOC exam pass rate. Amato-Cole Educational Services Preparatory Workshop The current study is the first to investigate the relationship between ACES Test 2 scores and first-time BOC exam pass rates. The ACES Preparatory Workshop was designed for students to learn test-taking strategies specific to the BOC exam, to identify content weaknesses, and to help with test anxiety. In nursing education, the HESI exit exam has been widely used as a predictor of NCLEX-RN examination first-time pass rates. Eight HESI validity studies indicated for nursing students who scored 900 or higher, the HESI highly predicted passing the NCLEXRN exam (96.36% to 99.16%) (Zweighaft, 2015). Given the vast research supporting the HESI examination for NCLEX-RN exam pass rates, there is some debate on the use of repeated testing to obtain the needed predictive score, and there is further debate on the second and third forms used and their predictive value on NCLEX-RN pass rates (Sosa & Sethares, 2015). In the current study, the second of the two ACES Tests administered as part of the ACES Preparatory Workshop was utilized for analysis. Typically, most students see an increase in scores after the first exam due to testing strategies emphasized at the beginning of the ACES Preparatory Workshop and familiarity with the examination format. Follow-up testing and alternate test versions were not examined in the current study and may need to be considered when making BOC exam readiness decisions. While only three programs were sampled, the ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 37 current study finds similarities of the ACES Test 2 results to nursings use of the HESI exit exam when providing program administrators and students direction when preparing for the BOC exam. While there is evidence to support ACES in attempting to reduce test anxiety (Breitbach et al., 2013), the current study provides program directors guidance to determine whether a student is ready to take the BOC exam, when examining scores from the ACES Preparatory Workshop. College Readiness Metrics and Final Program Grades Platt et al. (2001) found a correlation between the admissions metric of SAT scores with program academic success, finding SAT verbal scores had value in predicting academic success (college GPA) when coupled with high school GPA. While this study did not compare SAT metrics to first-time BOC exam pass rates, the findings were considered useful for admissions criteria in several healthcare programs: athletic training, physical therapy, occupational therapy, and physician assistant (Platt et al., 2001). Similarly, Hobson (2019) found that both verbal and math SAT scores were accurate in predicting first-time BOC exam pass rates. Harrelson et al. (1997) found ACT composite scores were able to predict BOC exam first-time pass rates; however, ACT composite scores were only able to predict BOC exam first-time pass rates when inclusive of overall academic, athletic training, and academic minor GPA. In the current study, standardized admission test scores were reported as SAT and were converted to an ACT equivalent. Regarding relating ACT scores to BOC exam or program success, the current study found ACT scores between the groups to be statistically significant but had a small effect size indicating the ACT scores provided little clinical meaning in this study. Moreover, ACT composite scores were not significant in the overall prediction model and perhaps should be used with caution when used to predict BOC exam outcomes. While SAT ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 38 verbal and math scores were relevant for BOC exam success, it might be prudent to compare ACT sub-scores to determine if there are similarities to the findings regarding SAT scores. While the current study focused on factors of success (first-time BOC exam pass rates) for undergraduate professional athletic training programs, it was noted that the GRE is used by many graduate athletic training programs for student admission. A review of several studies in athletic training and physical therapy found the GRE to produce mixed results in predicting academic success (GPA) or success on standardized board exams (first-time pass rates). Academic and or board exam success have been linked to GRE quantitative and or GRE verbal scores (Bayliss et al., 2017; Bruce et al., 2016;Thieman et al., 2003). Other studies did not find GRE scores (quantitative or verbal) to be significant factors regarding first-time exam pass rates on respective professional board exams (Keskula, 1995; Kuhn & Parrott, 2017; Riddle et al., 2009). As athletic training programs transition from bachelors to masters degrees, GRE scores should be investigated more thoroughly as a measure to determine student admission. Athletic training studies have correlated GPA with first-time BOC exam pass rates. Many of the studies were prior to the current BOC exam format; yet, these studies found GPA to be a predictor of BOC exam outcomes (Harrelson et al., 1997; Keskula, Sammarone, & Perrin, 1995). One study found that overall program GPA, athletic training GPA, and academic minor GPA were the strongest predictors of first-time BOC exam pass rates (Harrelson et al. 1997). When comparing clinical hours and program GPA, it was noted that the overall program GPA had the highest predictive value of first-time BOC exam pass rates. (Middlemas et al., 2001). In a study involving graduate athletic training students, it was noted that a student's admission GPA did not significantly predict BOC exam outcomes where the final masters program GPA did have a significant relationship with first-time BOC exam pass rates (Murray, 2014). A study ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 39 investigating graduate program GPA at the end of the first year of the athletic training program indicated students whose GPA was 3.45 had almost twice the probability of passing the BOC exam on the first attempt versus students with a GPA of < 3.45 (Bruce et al., 2016). Similarly, a study involving undergraduate students attributed the final program GPA in undergraduate ATPs to be significant predictors of BOC exam first-time pass rates (Cripps et al., 2015). Other studies indicated athletic training pre-admission GPA was one of the best predictors of program success (Lawrance & Lauber, 2015) and success on BOC exam first-time pass rates (Platt et al., 2001). Investigation into GPA and first-time board pass rate success in other healthcare education programs (physical therapy, occupational therapy, physician assistant) found similar results when comparing first-time pass rates on their respective board examinations (Bayliss et al., 2017; Hocking & Piepenbrock, 2010; Huhn & Parrot, 2017; Luedtke-Hoffmann et al., 2012; MacArthur & Randall, 2015; Riddle et al., 2009). In the current study, evidence points to the use of the final cumulative GPA when evaluating predictors of first-time BOC pass rates. Healthcare education programs may benefit by considering GPA as an admissions metric. However, it is perhaps more critical to evaluate the student's academic progress early and throughout the program. Being able to identify students progress early in the program will help mitigate academic deficiencies before taking standardized board examinations. Institutional Factors When the BOC Practice Analysis was distributed to students in the junior or senior year, those students were 3.13 times more likely to pass the BOC exam on the first attempt than students who were introduced to the BOC Practice Analysis in their sophomore year. The gap in time between a students sophomore year and the beginning of study preparation during their ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 40 senior year and increased maturity level between sophomores as compared to juniors and seniors may account for the increased likelihood of passing the BOC exam on the first attempt. Students in programs with three faculty members did have a 3.5 times greater chance of passing the BOC exam on the first attempt than students with only two faculty members. A possible reason for this outcome might be the strain on faculty when faced with the task of administering the required competencies for the CAATE approved program. The 2020 standards for professional programs require a minimum of three full-time core faculty (CAATE, 2019c). This study supports the standard requiring three full-time core faculty members. Age, Gender, and Primary Language Students in this study had a median age of 23 years, with age ranging from 19-30 years. The median age of those who passed the BOC exam on the first attempt was 23 years and 22 years for those who did not pass the BOC exam on the first attempt. The comparison between age and first-time BOC exam pass rates indicated the difference was not statistically significant. It would be reasonable to expect that the nonsignificant findings in this study were the result of professional undergraduate ATPs typically enrolling a traditional high school age student, while professional masters programs would have more age disparity by attracting an older student consistent with post-graduates and non-traditional students. Age as a factor for success on the BOC exam was one of many factors identified when considering the transition of the professional degree in athletic training to the masters level. A comparison of undergraduate and masters programs in athletic training from 2010-2012 found BOC exam pass rates higher in those graduating from masters programs, even in institutions with both undergraduate and masters level athletic training programs (NATA, 2013). It is thought that graduate students tend to be more mature and more apt to be self-directed, independent learners, and better able to apply ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 41 critical thinking skills. Given the higher propensity for graduate students to possess these traits, age, and maturity would likely account for the increased first-time BOC exam pass rates (NATA, 2013). There was no significant difference in gender between the two groups. While the BOC does not publish the gender of test-takers, data for this study indicated a higher number of females (64.3%) to males (35.7 %) compared to NATA gender membership data. National Athletic Trainers Association gender membership in 2018 was 56% for females and 44% for males (NATA, 2019). The primary language could not be compared between students who passed the BOC exam on the first attempt and with those who did not pass the BOC exam on the first attempt due to programs reporting that only four students spoke English as a second language. No research was found regarding the effect of primary language on first-time BOC exam pass rates. However, it was indicated as problematic in one study involving nursing students and the NCLEX exam where nursing students who spoke English as a second language passed the NCLEX exam at a lower rate than did those who spoke only English (O'Neil et al., 2006). Board of Certification Study Materials Athletic training programs typically utilize two forms of study guides when preparing students for the BOC exam. One is developed by the BOC for public consumption, and the other is a program developed study guide. Board exam preparation consisting of the number of hours, type of commercial study guide, and practice examinations found no correlation in first-time pass rates in physical therapy (Crawley et al., 2015). The current study comparing BOC exam firsttime pass rates when using a program-specific (non-BOC) and a BOC approved study guide contradicted findings from occupational therapy (Breen-Franklin, 2017). Breen-Franklin (2017) ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 42 found no significant difference regarding types of study guides used; whereas, the current study suggested the use of a non-BOC approved study guide placed students at a higher risk of failing the BOC exam. Individual programs may need to carefully investigate the validity of commercially produced study guides within their respective professions and pay particular attention to individual student deficiencies when choosing to produce their study guide for BOC exam preparation. Limitations and Barriers of Study A low program response rate limits this study. Initial contact was made through phone or email by the investigator for all six schools chosen for the study, and repeated follow-up was performed using the same medium. However, while data gathered was ample (N = 210), only three of the six programs contacted provided information for the study. A likely cause of the low program response rate is study timing. The data collection took place in the spring semester, a busy time of the year for many program directors as they prepare for year-end academic obligations. Many program directors are also currently engaged in the transition from bachelors degrees to CAATE mandated masters degree in athletic training. In response to the mandated transition, many program directors have been required to return to the classroom to pursue terminal degrees to meet the requirements of both regional accreditation and CAATE. Because only three schools participated, it was noted that the geographical and Carnegie classification was limited in scope and may not represent athletic training programs as a whole. It was also noted that many program directors have not kept adequate student records or have limited access to this data. One program was able to complete much of the Excel datasheet; however, it was unable to provide an age for their student population and another program that agreed to participate later realized access to data was unobtainable. While CAATE requires ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 43 program directors to log program and student information into an e-accreditation portal, it does not require the documentation of age, ACT/SAT scores, cumulative GPA, or other specific programmatic indicators. Tracking this information in this portal may be helpful for those institutions which do not have access to other forms of data gathering and dissemination. This study was also limited in the composition of the participants. Undergraduate ATPs by nature admit students within the same age demographic. This study revealed a median age of 23 years (N = 87), a typical age for a student graduating from an undergraduate program. This information may not be generalizable to the master's program demographics, which tend to attract a more diverse age group. English as a students secondary language was limited in this study to four students from one program; therefore, conclusions about the effect students' primary language could have on first-time BOC exam pass rates could not be made. Recommendations for Future Research The significance mentioned above in this study concerning English as a second language, and students who passed the BOC exam on the first attempt would suggest further examination. The recommendation would be to conduct a similar study where a broad range of programs can be sampled to get a better picture of whether English as a second language has an effect on firsttime BOC exam pass rates. While evidence in this current study does not suggest an internal problem with the BOC exam, it may give program directors valuable information regarding study strategies to employ, and it may raise the awareness of BOC officials when considering methods of test administration. Statistics on ethnic diversity are recorded by the NATA; however, data on English as a second language and BOC exam pass rates could not be found. As the NATA expands its international reach, programs may face more first-time BOC exam pass rate complications with an anticipated increase in an international student population who speak ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 44 English as a second language. More research in this area may provide valuable insight into the factors related to success and failure on the BOC exam in this population. Program directors in this study were surveyed to see whether they used BOC approved study materials or if they manufactured their own BOC study materials. The sample size was small, and all three programs indicated they used both forms of study materials for students before taking the BOC exam. A qualitative study identifying what study strategies students perceive as the most beneficial for first-time BOC pass rates may help program directors identify additional study strategies for the BOC examination preparation. This study revealed a significant positive correlation between ACES Test 2 scores and passing the BOC exam on the first attempt. In light of the results, an opportunity exists to expand this study to include a broader sample of programs. Perhaps demographics and ACES Test 2 scores data could be collected in a collaborative effort between ACES and the BOC at the beginning of the ACES Preparatory Workshop with follow-up gaining access from the BOC on matching BOC exam pass and fail rates. Collecting data in this manner would be beneficial in accessing samples from over 25 ACES testing sites in all geographical regions of the U.S. Information gained from a larger sample size may give program directors a better picture of how to best use the ACES Preparatory Workshop scores for assessment data and students preparation in advance of taking the BOC exam. Conclusion This study examined variables of ACES Preparatory Workshop Test 2 scores, ACT scores, final cumulative GPA, BOC study guide use, and non-BOC study guide use, program student enrollment, Carnegie classification, BOC Practice Analysis, number of core athletic training faculty, age, gender, and English as a second language and found final program GPA, ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 45 timing of distributing the BOC Practice Analysis, and ACES Test 2 scores to be significant factors in passing the BOC exam on the first attempt. Age and gender were not found to be significant factors for success on the BOC exam on the first attempt. The timing of the BOC Practice Analysis did reveal a significance in the logistic regression predictive model. Students presented with the BOC Practice Analysis in their junior or senior year were over 3 times more likely to pass the BOC exam as compared to sophomore year distribution. The ACES Preparatory Workshop was designed to be administered in a student's senior year to allow enough time to remediate before taking the BOC exam. However, several schools have the ACES Preparatory Workshop administered in the junior year. In this study, two programs hosted the ACES Preparatory Workshop in the student's junior year while the third program did so in the student's senior year. Amato-Cole Educational Services distributed the BOC Practice Analysis during the workshop to identify student areas of weakness within the domains of the BOC exam. Programs may want to be aware of students who are presented with the BOC Practice Analysis before taking the ACES Preparatory Workshop and may want to reemphasize the importance of reviewing the BOC Practice Analysis when beginning to prepare for the BOC exam in the senior year. Relying on the early distribution of the BOC Practice Analysis for the primary mode of BOC exam study may give a false sense of security if not utilized closer to taking the BOC exam. It is imperative that program directors utilize assessment data to demonstrate a level of program and student success and attract students in a competitive market. Passing the BOC exam on the first attempt is one of the significant indicators of success. As master's programs become the norm, program directors in all programs are looking to admit students with the best chance of succeeding on the BOC examination. Some programs serve a student who enters as a first- ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 46 generation college student eager to succeed. For those institutions, ensuring success is key to program survival, especially when student selectivity is not always an option. Programs of this nature need to review their program data and consider many factors such as admission and program GPA, and board preparation assessment examinations when devising a comprehensive plan to BOC examination preparation. Some program directors have begun to utilize the services of the ACES Preparatory Workshop to help identify areas of remediation before taking the BOC exam. This study revealed that the final cumulative GPA, coupled with an ACES Test 2 score, may provide valuable data when preparing for the BOC exam and ensuring program success. Students with lower program GPA and ACES Test 2 scores may need additional intervention and instruction. As the profession of athletic training increases in demand (23% between 2016 -2026) (Bureau of Labor Statistics, 2019), ensuring program and student success is one of the keys to maintaining a presence in today's healthcare system. ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 47 References ACT. (2019). ACT/SAT concordance - scores - The ACT test. Retrieved June 2, 2019, from http://www.act.org/content/act/en/products-and-services/the-act/scores/act-satconcordance.html Bayliss, J., Thomas, R., & Eifert-Mangine, M. (2017). Pilot study: What measures predict the first-time pass rate on the national physical therapy examination. Internet Journal of Allied Health Sciences and Practice, 15(4), 1-12. Retrieved from nsuworks.nova.edu/ijahsp Board of Certification. (2007). The BOC exam: The first 40 years. A tribute to our volunteers. Author. Board of Certification. (2019). Exam reports. Retrieved June 23, 2019, from https://bocatc.org/news#exam-reports-summary Box, G. E., & Tidwell, P. W. (1962). Transformation of the independent variables. Technometrics, 4, 531-550. Breen-Franklin, A. (2017). Enabling student success for the NBCOT exam: A pilot education program. Occupational Therapy in Health Care, 31(4), 358-364. Breitbach, A. P., Downey, D. L., & Frager, A. J. (2013). The relationship between candidate psychological factors and first-attempt pass rate on the Board of Certification examination. Athletic training Education Journal, 8(1-2), 10-16.doi:10.4085/08010210 Brown, G., Imel, B., Nelson, A., Hale, O., & Jansen, N. (2013). Correlation between PANCE performance, physician assistant program grade point average, and selection criteria. The Journal of Physician Assistant Education, 24(1), 42-44. Brown, C. D., Potteiger, K., Keeley, K., Lundgren, A., & David, S. (2017). A review of undergraduate athletic training program entrance requirements. Internet Journal of Allied ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 48 Health Sciences and Practice, 15(1), 1-10. Retrieved from http:/nsuworks.nova.edu/vol15/issl/4. Bruce, S., Crawford, E., Wilkerson, G., Rausch, D., Dale, R., & Harris, M. (2016). Prediction modeling for academic success in professional master's athletic training programs. Athletic training Education Journal, 11(4), 194-207. doi:10.4085/1104194 Bureau of Labor Statistics. (2019, April 12). Athletic trainers: Occupational outlook handbook: U.S. bureau of labor statistics. Retrieved from https://www.bls.gov/ooh/healthcare/athletic-trainers.htm Chen, S., & Voyles, D. (2013). HESI admission assessment scores: Predicting student success. Journal of Professional Nursing, 29(2), S32-S37. doi:10.1016/j.profnurs.2012.06.008 Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. Cook, C. E., Landry, M. D., Covington, J. K., McCallum, C., & Engelhard, C. (2015). Scholarly research productivity is not related to higher three-year licensure pass rates for physical therapy academic programs. BMC Medical Education, 15(1), 1-7. doi:10.1186/s12909015-0431-1 Commission on Accreditation of Athletic training Education. (2012). Standards for the accreditation of professional athletic training programs. Retrieved from https://caate.net/wp-content/uploads/2018/02/2012-Professional-Standards_.pdf Commission on Accreditation of Athletic training Education. (2014). Historical overview. Retrieved June 6, 2019, from https://caate.net/historical-overview/ Commission on Accreditation of Athletic training Education. (2015). Joint statement from the strategic alliance BOC, CAATE, NATA and NATA Foundation. Retrieved from ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 49 CAATE.net: https://caate.net/wp-content/uploads/2015/05/Strategic-Alliance-CCATEemail-pdf.pdf Commission on Accreditation of Athletic training Education (2016a). 2015-2016 CAATE analytics report. Retrieved from CAATE.net: https://caate.net/wpcontent/uploads/2017/09/2015-16-CAATE-Analytics-Report_VF.pdf Commission on Accreditation of Athletic training Education (2016b). Official communication regarding upcoming review and actions related to professional standard 11. Retrieved from CAATE.net: https://caate.net/wp-content/uploads/2016/02/CAATE-ProfessionalStandard-11.pdf Commission on Accreditation of Athletic training Education. (2019a). Search for accredited programs. Retrieved July 7, 2019, from https://caate.net/search-for-accredited-program/ Commission on Accreditation of Athletic training Education (2019b). Program outcomes. Retrieved from https://caate.net/program-outcomes/ Commission on Accreditation of Athletic training Education. (2019c). Commission on accreditation of athletic training education 2020 standards for accreditation of professional athletic training programs masters degree programs. Retrieved from https://caate.net/wp-content/uploads/2019/02/2020-Standards-Final-2-20-2019.pdf Cottrell, R., & McKenzie, J. (2011). Health promotion & education research methods: Using the five-chapter thesis/dissertation model (2nd ed.). Sudbury, MA: Jones and Bartlett Publications. Crawley, R. (2015). The relationship between study preparation and first attempt pass rate on the national physical therapy examination. Journal of the National Society of Allied Health, ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 50 12(1), 45-52. Retrieved from http://nsah.org/wp-content/uploads/2016/09/NSAH-Journal2015.pdf Crouch, S. J. (2015). Predicting success in nursing programs. Journal of College Teaching & Learning, 12(1), 45-54. doi:10.19030/tlc.v12i1.9069 Delforge, G. D., & Behnke, R. S. (1999). The history and evolution of athletic training education in the United States. Journal of Athletic training, 34(1), 53-61. Dockter, M. (2001). An analysis of physical therapy preadmission factors on academic success and success on the national licensing examination. Journal of Physical Therapy Education, 15(1), 60-64. doi:10.1097/00001416-200101000-00010 Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. Field, A. (2017). Discovering statistics using IBM SPSS statistics: North American edition (5th ed.). Thousand Oaks, CA: SAGE. Grace, P. (1999). Milestones in athletic trainer certification. Journal of Athletic training, 34(3), 285-291. Harrelson, G. L., Gallaspy, J. B., Knight, H. V., & Leaver-Dunn, D. (1997). Predictors of success on the NATABOC certification examination. Journal of Athletic training, 32(4), 323327. Hawkins, J., McLoda, T., & Stanek, J. (2015). A subjective and objective process for athletic training student selection. Athletic training Education Journal, 10(2), 183-189. doi:10.4085/1002183 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 51 Herrera, C., & Blair, J. (2015). Predicting success in nursing programs. Research in Higher Education Journal, 28(5), 1-8. Hinderer, K. A., DiBartolo, M. C., & Walsh, C. M. (2014). HESI admission assessment (A2) examination scores, program progression, and NCLEX-RN success in baccalaureate nursing: An exploratory study of dependable academic indicators of success. Journal of Professional Nursing, 30(5), 436-442. doi:10.1016/j.profnurs.2014.01.007 Hobson, Z. R. (2019). Accuracy of athletic training student performance factors for determining BOC exam first-time pass outcomes (Master's thesis). Retrieved from https://etda.libraries.psu.edu/catalog?utf8=%E2%9C%93&search_field=all_fields&q=Ho bson Hocking, J., & Piepenbrock, K. (2010). Predictive ability of the graduate record examination and its usage across physician assistant programs. Journal of Physician Assistant Education, 21(4), 18-22. Huhn, K., & Parrott, S. J. (2017). Exploration of relationships among the Health Sciences Reasoning Test, the National Physical Therapy Licensing Examination, and cognitive admission variables. Journal of Physical Therapy Education, 31(1), 7-13. doi:10.1097/00001416-201731010-00004 Institute of Medicine. (2001). Crossing the quality chasm: A new healthcare system for the 21st century. summary. Retrieved from National Academy of Sciences website: http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2001/Crossingthe-Quality-Chasm/Quality%20Chasm%202001%20%20report%20brief.pdf ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 52 Keskula, D. R., Sammarone, P. G., & Perrin, D. H. (1995). Prediction of academic achievement in an NATA-approved graduate athletic training education program. Journal of Athletic training, 30(1), 5556. Kosmahl, E. M. (2005). Factors related to physical therapist license examination scores. Journal of Physical Therapy Education, 19(2), 52-56. doi:10.1097/00001416-200507000-00007 Lawrence, S. E., & Lauber, C. A. (2015). Do pre-admission and athletic training program standards predict student academic success and first-time pass rate on the Board of Certification exam? Athletic training Education Journal, 10(1), s15. Retrieved from https://natajournals.org/doi/pdf/10.4085/1947-380X-10.sp1.S-1 Luedtke-Hoffmann, K., Dillon, L., Utsey, C., & Tomaka, J. (2012). Is there a relationship between performance during physical therapist clinical education and scores on the National Physical Therapy Examination (NPTE)? Journal of Physical Therapy Education, 26(2), 41-49. doi:10.1097/00001416-201201000-00008 MacArther, J., & Randall, J. (2015). Predictive factors of successful outcomes for occupational therapy students (Master's thesis). Retrieved from https://soundideas.pugetsound.edu/ms_occ_therapy/105/ Middlemas, D. A., Manning, J. M., Gazzillo, L. M., & Young, J. (2001). Predicting performance on the National Athletic Trainers' Association Board of Certification Examination from grade point average and number of clinical hours. Journal of Athletic training, 36(2), 136140. Murray, M. E. (2014). Predictors of success in entry-level master's degree programs in athletic training (Doctoral dissertation). Retrieved from https://scholarship.shu.edu/cgi/viewcontent.cgi?referer=https://scholar.google.com/schola ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 53 r?hl=en&as_sdt=0%2C50&q=predictors+of+BOC+exam+for+entrylevel+masters+of+athletic+training+progams&btnG=&httpsredir=1&article=2988&conte xt=dissertations National Postsecondary Education Cooperative. (2009). Information required to be disclosed under the higher education act of 1965: Suggestions for dissemination (updated). NCES.ed.gov. Retrieved from https://nces.ed.gov/pubs2010/2010831rev.pdf Novalis, S. D., Cyranowski, J. M., & Dolhi, C. D. (2017). Passing the NBCOT examination: Preadmission, academic, and fieldwork factors. The Open Journal of Occupational Therapy, 5(4), 1-16. doi:10.15453/2168-6408.1341 National Athletic Trainers Association (2013). Professional education in athletic training: An examination of the professional degree level. NATA.org, 125. Retrieved from: https://www.nata.org//sites/default/files/The_Professional_Degree_in_Athletic_Training. pdf NATA. (2017). Education overview. Retrieved August 18, 2019, from https://www.nata.org/about/athletic-training/education-overview NATA. (2019). Membership statistics. Retrieved June 2, 2019, from https://members.nata.org/members1/documents/membstats/2018EOY-stats.htm O'Neil, T. R., Marks, C., & Lie, W. (2006). Assessing the impact of English as a second language status on licensure examinations. CLEAR Exam Review, 14-18. Ostrowski, J. L., & Marshall, B. (2015). Master's level professional athletic training programs: Program characteristics, graduation requirements, and outcome measures. Athletic training Education Journal, 10(1), 25-31. doi:10.4085/100125 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 54 Platt, L. S., Turocy, P. S., & McGlumphy, B. E. (2001). Preadmission criteria as predictors of academic success in entry-level athletic training and other allied health educational programs. Journal of athletic training, 36(2), 141144. Riddle, D. L., Utzman, R. R., Jewell, D. V., Pearson, S., & Kong, X. (2009). Academic difficulty and program-level variables predict performance on the National Physical Therapy Examination for licensure: A population-based cohort study. Physical Therapy, 89(11), 1182-1191. doi:10.2522/ptj.20080400 Sosa, M., & Sethares, K. A. (2015). An integrative review of the use and outcomes of HESI testing in baccalaureate nursing programs. Nursing Education Perspectives, 36(4), 237243. doi:10.5480/14-1515 Tabachnick, B. G., & Fidell, L. S. (2014). Using multivariate statistics (6th ed.). Essex, UK: Pearson. Thieman, T. J., Weddle, M. L., & Moore, M. A. (2003). Predicting academic, clinical, and licensure examination performance in a professional (entry-level) masters degree program in physical therapy. Journal of Physical Therapy Education, 17(2), 32-37. doi:10.1097/00001416-200307000-00005 Turocy, P. S., Comfort, R. E., Perrin, D. H., & Gieck, J. H. (2000). Clinical experiences are not predictive of outcomes on the NATABOC examination. Journal of Athletic training, 35(1), 7075. United States Department of Labor. (2018). Occupational outlook handbook. Retrieved from Bureau of Labor Statistics website: https://www.bls.gov/ooh/healthcare/registerednurses.htm ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION Utzman, R. R., Riddle, D. L., & Jewell, D. V. (2007). Use of demographic and quantitative admissions data to predict academic difficulty among professional physical therapist students. Physical Therapy, 87(9), 1164-1180. doi:10.2522/ptj.20060221 Zweighaft, E. L. (2013). Impact of HESI specialty exams: The ninth HESI exit exam validity study. Journal of Professional Nursing, 29(2), S10-S16. doi:10.1016/j.profnurs.2012.06.011 55 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 56 Table 1 Description of Athletic training Programs Program 1 Program 2 Program 3 26 34 30 Carnegie classification Bachelor Doctoral Masters BOC Practice Analysis Junior Senior Sophomore Core athletic training faculty 3 3 2 Standard 11 non-compliance 2013-2014 NA 2016-2019 BOC study guide Yes Yes Yes Non-BOC study guide Yes No Yes Program student enrollment Note. BOC = Board of Certification, NA = Not applicable due to being compliant ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 57 Table 2 Student Characteristics by Program Median (IQR) age (years) Program 1 Program 2 Program 3 (N = 76) (N = 98) (N = 36) 23.00 (1.00) N (%) 22..00 (1.0) N (%) N (%) BOC p <.001 Pass 58 (76.3%) 89 (90.8%) 22 (61.1%) Fail 18 (23.7%) 9 (9.2%) 14 (38.9%) Gender .890 Male 26 (34.2%) 35 (35.7%) 14 (38.9%) Female 50 (65.8%) 63 (64.3%) 22 (61.1%) 72 (94.7%) 98 (100%) 36(100%) 4 (5.3%) 0 (0%) 0 (0%) Primary language English Non-English Note. IQR = interquartile range, BOC = Board of Certification ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 58 Table 3 Outcomes by Students who Passed and Failed the Board of Certification ACT Scores Final cumulative GPA ACESa BOC Passed BOC Failed (N = 159) (N = 40) Mdn (IQR) Mdn (IQR) p 24.00 (4.00) 22.00 (2.00) .004 3.73 3.42 < .001 81.04 (9.19) 72.07 (7.95) < .001 N (%) N (%) Non-BOC Guide < .001 No 89 (90.8%) 9(9.2%) Yes 80(80.4%) 32(28.6%) BOC Practice Analysis < .001 Sophomore 22 (61.1%) 14(38.9%) Junior 58(76.3%) 18(23.7%) Senior 89(90.8%) 9(9.2%) Core Faculty < .001 2 22(61.1%) 14(38.9%) 3 147(87.5%) 27(15.5%) Note. IQR = interquartile range, BOC = Board of Certification, ACT =American college testing, SAT = scholastic aptitude test, GPA = grade point average, ACES = Amato-Coles educational services a Means and standard deviations reported. ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 59 Table 4 Logistic Regression Predicting Likelihood of Passing the BOC based on Institutional Factors, Undergraduate Assessments, and Student Characteristics SE Wald p Odds 95% CI Ratio Lower Upper Final cumulative GPA -2.66 0.77 11.88 .001 14.29 3.15 66.67 ACES Test score -.116 0.03 15.76 < .001 1.12 1.06 1.19 BOC Practice Analysis 1.14 0.49 5.45 .020 3.13 1.2 8.13 Constant 16.62 3.16 27.76 < .001 Note. ACES = Amato-Cole educational services; BOC = Board of Certification; a BOC Practice Analysis is for sophomore compared to junior and senior. ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION Appendix A Excel Data Spread Sheet 60 ATHLETIC TRAINERS BOARD OF CERTIFICATION EXAMINATION 61  ...

... EXAMINING THE BEHAVIORAL CHARACTERISTICS OF CHILDREN AT-RISK OF SENSORY MODULATION DEFICITS AND AUDITORY PROCESSING DEFICITS Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Barbara Warnock, M.S., OTR/L Copyright November 8, 2019 By: Barbara Warnock, M.S., OTR/L All rights reserved Approved by: Stephanie A. Miller, PT, PhD Committee Chair Sarah Schoen, PhD, OTR/L Committee Member Doug Darbro, PhD Committee Member Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis Running head: CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICITS Examining the Behavioral Characteristics of Children At-Risk of Sensory Modulation Deficits and Auditory Processing Deficits Barbara Warnock University of Indianapolis 1 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICITS 2 Abstract Early childhood is a critical time for expediential neurological growth that provides the foundation for learning, playing, self-care, and social engagement. Deficits in processing auditory input beyond peripheral hearing can impede this early neurological foundation in children. Deficits in sensory modulation (SMD) and auditory processing (APD) are two disorders that appear to have similar auditory behavioral characteristics but have not been compared in the literature. The purpose of this study was to determine the number of children atrisk of these two disorders, determine if there was a relationship between the characteristics of children at-risk of these two disorders, and determine the predictors of each condition. A quantitative study using a non-experimental design was used to answer the research questions. Retrospective data from the Short Sensory Profile and the Tests for Auditory Processing Disorders were obtained from 309 children ages 5 years through 11 years, 11 months that were referred for an occupational therapy evaluation at a private clinic in western United States. Results indicated that 52.4% of the cases were at-risk of SMD and 5.9% of the cases were at-risk of APD. Those at-risk of both SMD and APD were 3.4%. There was a significant relationship between subjects having Attention Deficits Hyperactivity Disorder and being at-risk of SMD. There was also a significant relationship between having Autism Spectrum Disorder and being at-risk of APD. The odds of being at-risk of SMD were 2.75 times higher if the subjects had ADHD and the odds of being at-risk of APD were 5.12 time higher if the subjects had ASD. Results indicated that for this sample, the majority of children have auditory deficits within SMD and not APD. A very small number of cases had co-morbidity of APD and SMD. Professionals need to refer children with ADHD for evaluation for SMD and children with ASD for APD. Additional research is needed to examine these finding in the general population. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 3 Table of Contents Title Page.1 Abstract2 Chapter 1: Introduction5 Chapter 2: Literature Review...7 The Importance of Screening At-Risk Children..8 Sensory Modulation Deficits.....10 Screening for Sensory Modulation Deficits...12 Auditory Processing Deficits ....13 Screening for Auditory Processing Deficits..14 Purpose of Study....15 Clinical Relevance.16 Chapter 3: Method 17 Study Design..17 Participants.17 Data Collection..18 Operational definition of variables....18 Instruments.19 Short Sensory Profile.19 Children: Tests for Auditory Processing Disorders.. 20 Procedures..21 Data Analysis.21 Chapter 4: Results..25 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 4 Demographic Characteristics.25 At-Risk of Sensory Modulation Disorder..26 At Risk of Auditory Processing Disorder..27 At Risk of Both SMD and APD Conditions..28 Relationship Between Patient Characteristics When Identified as Being at Risk of SMD, APD or Both Conditions30 Prediction of the odds of being at risk for SMD, APD, or both conditions...32 Prediction for SMD (Model 1)...32 Prediction for APD (Model 2)...36 Prediction for SMD and APD (Both) (Model 3)...40 Summary of Results...40 Chapter 5: Discussion....41 Limitations.....48 Clinical Relevance and Future Research...49 References..52 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 5 Examining the Behavioral Characteristics of Children At-Risk of Sensory Modulation Deficits and for Auditory Processing Deficits Sensory modulation disorder (SMD) is a neurological condition that encompasses an inefficiency in regulating and responding appropriately to sensory experiences resulting in impaired participation in daily life (Miller, Anzalone, Lane, Cermak, & Osten, 2007). Sensory modulation disorder is a subtype of the over-arching disorder identified as sensory processing disorder. Sensory modulation deficits specific to the auditory system occur when sound overregisters, under-registers, or causes a child to seek constant input, also referred to as sensory craving (Miller et al., 2007). Modulation deficits within the auditory system differs from other auditory deficits such as auditory processing disorder, which is the inability to identify the fine details of auditory sensation, such as hearing the difference between bad and lad (Kuczynski & Kolakowsky-Hayner, 2011). While these described auditory deficits have distinct definitions, in children they can present with similar behavioral expressions, such as inattention in a noisy environment (Chermak, Tucker, & Seikel, 2002; Dunn, 1999; Ferre, 2015). It is unknown if these two auditory deficits are related and if so, how are they related. In clinical practice settings, it is typical for occupational therapists (OTs) to address sensory modulation deficits and for audiologists to focus on auditory processing deficits. While best practice calls for interprofessional collaboration in rehabilitation, it is more common for children with auditory deficits to be seen by only a single discipline (American Speech and Hearing Association [ASHA], 2005; Sharma, Purdy, & Kelly, 2012; Tazeau & Hamaguchi, 2013). Typically, audiologists evaluate children with auditory processing concerns with assessments, such as the Children: Tests for Auditory Processing Disorders, third edition (SCAN-3: C). This assessment measures various components of how auditory input is processed CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 6 beyond the peripheral auditory system (Keith, 2009). Children who have difficulty in their everyday activities, such as school, play and self-care due to sensory deficits are often referred to OTs. Occupational therapists use screening tools, such as the Short Sensory Profile (SSP), which measures behavioral responses to auditory sensations and other sensations that occurs in their natural environment (Dunn, 1999). Common practice of only one discipline addressing auditory deficits could put children at risk for not receiving appropriate intervention for all auditory deficits impacting their development. In addition, there is a lack of research that has examined the number of children who are at-risk for either or both sensory modulation deficits and auditory processing deficits that interfere with childhood development (de Wit et al., 2017; Tazeau & Hamaguchi, 2013). Therefore, the purpose of this study is to determine the number of children at-risk of a sensory modulation deficit and being at-risk of an auditory processing deficit in children referred for OT and if there are characteristics associated with being at-risk of one or both of these conditions. To meet this purpose, the following objectives were addressed. 1. To determine the number of children at-risk of sensory modulation deficits, at-risk of auditory processing deficit, or at-risk for both. 2. To determine if there is a relationship between patient characteristics and being at risk of one or both of these conditions. If warranted, determine predictors of each condition. Results from this study may be used to support a clearer understanding of the relationship among these two pediatric conditions and strengthen the ability to screen and refer these children appropriately. Results also have the potential to support therapists to design and implement treatments that are specific to the characteristics of identified deficits. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 7 Literature Review Sensory modulation deficits in the pediatric population can have a negative impact on childhood development (Ahn, Miller, Milberger, & McIntosh, 2004; Parham & Mailloux, 2015; Shapiro, 2016). Children may become overwhelmed and miss important auditory information in a noisy school environment causing poor academic performance (Aazh, Moore, & Prasher, 2011). Social events may be avoided due to the overwhelming sensory experience, impacting the ability to form and keep friendships (Ben-Sasson, Carter, & Briggs-Gowan, 2009; Cosbey, Johnston & Dunn, 2010). The National Center for Health Statistics reported that approximately 30% of all children identified with developmental concerns receive services prior to kindergarten (Rice et al., 2014; Salinas, Gutierrez, Garcell, & Hernandez-Montiel, 2013). Identifying developmental delays in and of itself can be challenging, relying on parent education, teachers, pediatricians, and other health professionals knowledge of appropriate developmental screening tools, and follow through by families to obtain needed services when recommended (Center for Disease Control and Prevention, 2019). Based upon the literature, it is unknown the current percentage of children at risk for sensory modulation and/or auditory processing and how they are being identified during the early childhood period of development. The most recent research that examined the prevalence of sensory modulation disorder was nearly fifteen years ago (Ahn et al., 2004). Based upon results of a survey completed by more than 700 parents of kindergarten children, approximately 5% had deficits in sensory modulation (Ahn et al., 2004). The prevalence of auditory processing disorder in children has primarily been studied by the United States and United Kingdom (American Academy of Audiology [AAA], 2018; Hind et al., 2011). The most recent research published on the prevalence of auditory processing disorder was in 2011 (Hind et al., 2011). Approximately 5.1% CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 8 of children referred to an audiologist due to difficulty listening in noisy environments had auditory processing deficits (AAA, 2018; Hind et al., 2011). The ratio of males to females with sensory processing disorder was found to be 2:1 (Chermak & Musiek, 1997; Fraiser, Goswami, & Conti-Ramsden, 2010; Palfery & Duff, 2007). There is limited research exploring gender difference in children with sensory modulation disorder. A study by Schoen, Miller, and Flanagan (2018) identified a higher number of boys to girls (78% to 22%) with sensory processing disorder in a pre-post retrospective study looking at the efficacy of treatment. Articles located that included gender difference in children identified with sensory modulation deficits were under the umbrella term sensory processing disorder. An older article of 20 years identified auditory processing disorder was more prevalent in boys at the rate of 2:1 (Chermak & Musiek, 1997). This study aims to better described prevalence, characteristics, and possible relationship between sensory modulation deficits and auditory processing deficits that is lacking in the literature. The Importance of Screening At-Risk Children Early childhood has been identified as the critical period for screening and identifying developmental deficits based upon the neuroplasticity that occurs at a rapid pace during this time (Gonzlez Salinas, Garcia Guierrez, Garcell, & Hernandez-Montiel, 2014; Mundkur, 2005). Neuroplasticity is influenced by a positive environment that presents opportunities for exploration, experience, and mastery specific to the needs of the child (Mundkur, 2005). An environment that is considered a poor fit for childrens individual differences can cause stress and hinder the needed neuroplasticity that supports development (Barrasso-Catanzaro & Eslinger, 2016; Miller, Reisman, McIntosh, & Simon, 2001). Individualized intervention that supports development, first requires appropriate screening and referrals to the necessary CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 9 professional services. Accurately screening children at risk for delay is dependent upon a thorough list of indicators and knowledge of overlapping behavioral characteristics among various developmental disorders (de Wit et al., 2017). Professionals are challenged when the symptoms presented by a child do not fit into one single condition, thus there can be variability in referral and diagnosing the condition. A case study by Russell, Norwich, and Gwernan-Jones (2012) followed a six-year-old child that had symptoms of learning difficulties, who was assessed by an interdisciplinary team and three different school psychologists. The child received three different diagnoses, revealing the inherent overlapping characteristics of conditions that challenge learning (Russell et al., 2012). Unless there are clearly defined characteristics of a disorder, professionals are not able to decipher between a single condition or co-morbidity. Studies addressing characteristics of auditory processing disorder and sensory modulation disorder with other pediatric disorders have occurred, but none have compared these two processing disorders. Characteristics of auditory processing disorder and attention deficits were studied by Gyldenkaerne, Dillon, Sharma, and Purdy (2014) concluding that APD and attention deficits co-exist in some children but were also independent conditions. A study examined the comorbidity of language and reading deficits in a group of children (N = 68) with auditory processing disorder (Sharma et al., 2012). Results indicated about half of the children had all three deficits, others had reading and auditory processing deficits, and others had language and auditory processing deficits. Stand-alone conditions were limited. When looking at co-morbidity in sensory modulation disorder, a study by Yochman, Alon-Beery, Scribman and Parush (2013) examined the ability to differentiate between sensory modulation disorder and attention deficit hyperactive disorder. Results indicated significant group differences in sensory responses but not CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 10 attention. Miller, Nielsen, and Schoen (2012) compared behavior and physiological aspects of children with attention deficit hyperactive disorder (ADHD) and SMD. Based on the results, the researchers indicated children with ADHD had more behavioral deficits with inattention and children with SMD had more deficits in sensory and emotional behaviors. Also, children with SMD had a greater degree of physiological/electrodermal response to sensory experiences than children with ADHD (Miller et al., 2012). Another study used a Sensory Challenge Protocol, with results indicating sensory modulation deficits in children with ADHD based upon increased levels of cortisol and electrodermal measurements (Lane, Reynolds & Thacker, 2010). These are a few examples of studies that have examined sensory modulation deficits and auditory processing disorder with other diagnoses, yet the author was not able to locate one study that differentiated between sensory modulation and auditory processing (Ghanizadeh, 2011; Pfeiffer, Daly Nicholls, & Gullo, 2014; Rogers, Hepburn, & Wehner, 2003; Tomchek & Dunn, 2007). The lack of this research may hinder early identification and appropriate treatment of children at risk for these disorders. Sensory Modulation Deficits When it comes to screening children at risk for sensory modulation deficits, this disorder is not included in the Diagnostic and Statistical Manual of Mental Disorders-5th edition (DSM5), which defines and classifies mental health conditions (Wikipedia, 2018). The current edition of Diagnostic Classification of Mental Health and Developmental Disorders of Infancy and Early Childhood (DC: 0-5) does include sensory processing disorder under Axis 1: Clinical Disorders and defines this as behaviors that reflect abnormal regulation of sensory input (Zero to Three, 2016). The educational system identifies sensory modulation disorder as a condition which impairs academic achievement as Other Health Impairment (Lucker, 2015; Ohio Department CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 11 of Education, 2014). Sensory modulation deficits occur when there is an inability to adapt and elicit an appropriate behavioral response comparative to the sensory experience (Miller et al., 2001). Responses may be influenced by task demand, intensity of sensory sensation, and duration of input (James, Miller, Schaaf, Nielsen, & Schoen, 2011). It is the combination of all these factors that may impact a childs reaction to sensory experiences. Children with efficient sensory modulation abilities are able to filter out undesirable or unimportant sensory information and focus on what is important (Miller et al., 2007). There are three subtypes of sensory modulation, including sensory over-responsivity (SOR), sensory under-responsivity (SUR), and sensory craving (SC) (James et al., 2007). When a child has difficulty managing the sensory demand, it may be due to the sensation perceived with too much intensity, referred to as SOR (Miller et al., 2007). When considering SOR in the auditory system, children may react to non-noxious sounds as threatening, painful, loud, and/or extreme agitation (Ben-Sasson et al., 2009; Tyler et al., 2014). Auditory SOR has been identified as co-occurring in various diagnoses, such as Autism Spectrum Disorder (63%) (Van Hulle, Schmidt, & Goldsmith, 2012), Williams syndrome (95-100%) (Aazh et al., 2011), ADHD and Fragile X (Reynolds & Lane, 2007). The prevalence of children at risk for sensory modulation deficits with auditory SOR is unknown and it is unknown if this condition overlaps with auditory processing deficit. Another behavioral response identified in some children with sensory modulation deficits is sensory under-responsivity (SUR) (Miller et al., 2007). This can be observed in children that do not respond or have a poor awareness to one or more sensation. Auditory SUR can present itself as children appearing not to hear their name called out, or the sound of a car coming toward them when running out to get a ball. The prevalence of children at risk for sensory modulation CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 12 deficits with auditory SUR has not been determined, nor if this condition overlaps with auditory processing. Sensory craving is when a sensation is continually sought out without a meaningful saturation of input (Miller et al., 2007). Auditory SC is observed when a child constantly makes audible sounds either vocally (humming, talking, singing, or other non-verbal sounds) or through the environment (tapping a pencil, tapping finger on a desk, or tapping foot on the floor). Similar to both SOR and SUR, the prevalence of children at risk for SC has not been explored or determined if it overlaps with auditory processing deficits. Screening for Sensory Modulation Deficits Various tools are used to screen sensory modulation deficits in practice. The Sensory Processing Measure (SPM) includes subtests for both sensory modulation and postural disorders (Parham et al., 2007). Touch Inventory for elementary school-aged children (TIE) measures SOR in the area of touch (Royeen & Fortune, 1990). Sensory Experiences Questionnaire 3rd ed. (SEQ) measures SUR and SOR in children with ASD (Ausderau & Baranek, 2013), and the Sensory Processing Three Dimensions Inventory (SP3D) measures SOR, SUR, and SC (Miller, Schoen & Mulligan, 2016). The Short Sensory Profile (SSP) is frequently used as a screening tool by OTs when measuring sensory modulation deficits (McIntosh et al., 1999). During its development, its intent was to provide professionals an appropriate screening tool to identify sensory modulation deficits and to be used for research (Dunn, 2008). The SSP is a 38-item inventory that uses a 5point Likert scale (Always-1 point, Frequently-2 points, Occasionally-3 points, Seldom-4 points, and Never-5 points) rating the frequencies of a childs behavioral response to sensory input. Higher scores indicate more typical responses to sensory experiences (McIntosh et al., 1999). This screening tool has been used with a variety of disabilities in research. These include autism CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 13 (Tomchek & Dunn, 2007), pediatric bipolar (Engel-Yeager et al., 2016), Phelan-McDermid syndrome (Mieses et al., 2016), behavioral problems (Gourley, Wind, Henninger, & Chinitz, 2013), fetal alcohol syndrome (Hansen & Jirikowic, 2013), prematurity (Crozier et al., 2016), among others. It has been found to discriminate between children with and those without sensory modulation difficulties (Dunn, 1999). The SSP items are mainly focused on behaviors that indicate SOR, with a few exceptions of SUR. Auditory items on the SSP related to sensitivity were used for this study (Dunn, 1999). Auditory Processing Deficits Auditory processing disorder is not included in the DSM-5, similar to sensory modulation disorder, nor the DC: 0-5 classification manual (Child Mind Institute, 2018; Zero to Three, 2016). The AAA (2010) and ASHA (2005) define auditory processing and its practice guidelines within their scope of practice. Auditory processing is the ability to recognize the fine details of spoken language, which is processed in the central auditory system. Deficits in auditory processing are not directly related to peripheral hearing loss, cognitive deficits, poor attention, or language disorder (ASHA, 2005). Specific auditory skills are related to auditory discrimination, binaural processing, and temporal processing. These three global areas involve the ability of an individual to lateralize and locate sound, order and integrate sound, and perform dichotic listening during competitive noisy situations. Auditory discrimination is the ability to analyze the fine details of sound, such as the phonemes of language. Binaural processing is the ability for both ears to work together to locate, attend, and listen to auditory sounds. Temporal pattern recognition is the ability to identify and separate the beginning and end of sounds presented is a sequence such as in a conversation (Ferre, 2015). Due to these many components of auditory CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 14 processing, distinguishing auditory processing disorder from other dysfunctions require meticulous and accurate diagnostic skills (Bellis, n.d.). To further understand auditory processing, five areas of listening are typically screened by audiologists for children demonstrating at-risk behaviors. These include auditory figureground, filtered words, competing words, and competing sentence (Keith, 2009; Ross-Swain, 2013). Auditory figure ground is the ability to process speech when in an environment with background sound. Filtering words is the ability to process distorted speech. Competing word is the ability to process speech when words are presented in each ear at the same time. Lastly, competing sentences is the ability to process and repeat back a sentence when unrelated sentences are presented in the left/right ears at the same time (Keith, 2009). Screening results determines if a global evaluation is deemed by the appropriate profession(s) (Chermak et al., 2002; de Wit et al., 2017). Screening for Auditory Processing Deficits The SCAN-3: C is the third edition of the childrens version of Tests for Auditory Processing Skills, which is standardized to screen and contribute to the diagnosing of auditory processing deficits. It is used with children ages 5 years 0 months to 12 years 11 months, measuring auditory figure-ground, filter words, competing words-directed ear, competing sentences, and ear advantage (Keith, 2009). Using an EBSCO search, this screening tool has been used in around 90 research studies and have focused on etiology, diagnosis, therapy, prognosis, and clinical prediction. There is no gold standard auditory processing screening or diagnostic tool but the SCAN-3: C and its prior versions have been widely used in research and is considered an evidence-based diagnostic test battery (Keith & Farah, 2013). CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 15 Purpose of Study Children may demonstrate early symptoms of SMD or APD as demonstrated by having difficulty processing sound at a very early age without having peripheral hearing loss. During infancy and toddler ages, behaviors such as a lack of response to sound, crying when in a crowded room, or lack of visual focus when competing sounds occur may indicate deficits in auditory processing or sensory modulation (Geffner, 2013; Williamson & Anzalone, 2001). Older children may have difficulty following verbal directions, mispronouncing words, or become aggressive when loud or unexpected sounds occur (Geffner, 2013; Tazeau & Hamaguchi, 2013). Many of these behaviors are described by parents and teachers when determining if further evaluations are needed. These behaviors are noted on screening tools used by both audiologists and occupational therapists. Confusion can occur when determining what professionals should be involved in screening and assessing these children secondary to overlapping symptoms or co-morbidity (Tazeau & Hamaguchi, 2013). Peripheral hearing loss is screened typically in school settings, primary care visits, or free community screening services (AAA, 2011). When there is no peripheral hearing loss, but children continue to have auditory deficits, further screening should occur to determine if a modulation or processing problem is occurring within the auditory system. Currently, there are no studies that have examined the same and/or different characteristics for SMD and APD, which can cause listening difficulties when there is no peripheral hearing loss. It is the purpose of this study to define these characteristics and evaluate the prevalence of children at risk for these two deficits. It is also the purpose of this study to determine if there is a relationship between these characteristics and being at risk for one or both of these disorders. If appropriate, determine the predictor factors for each of these conditions. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 16 Clinical Relevance When it comes to children with sensory modulation and/or auditory processing deficits, a single discipline is currently the primary practice approach used by health care professionals. It has been found that using an inter-professional approach to services has multiple benefits that outweigh the single discipline approach (Green & Johnson, 2015). These include empowering health professionals to make treatment recommendations, lessen the chance for missed deficits and miscommunication to client, provide a comprehensive view of client, promote affective and cost-efficient services, increase morale, and promote patient-center care (Education Management Solutions, 2017). As health and educational cost rises, an inter-professional approach to evaluation and treatment for children with auditory deficits can provide the efficiency needed to lessen the financial burden on families and health entities. The focus of occupational therapy is to promote health, well-being, and participation of daily activities that individuals find purposeful and meaningful (American Occupational Therapy Association [AOTA], 2014). The framework of the profession is carried out with the use of clinical reasoning, which includes an understanding of the scientific condition of the individual (Schell & Cervero, 1993). The clinical relevance of this study is to guide the early screening stage of services in order to make referrals to the most appropriate discipline for a more comprehensive evaluation. A clearer understanding of these two conditions can guide disciplines to develop and implement appropriate treatment according to the specific deficits. Each team member needs to understand their role on the team and ways to better collaborate. This research will provide the foundation for future research looking at the efficacy of treatment using an interprofessional team approach when working with children with deficits in the auditory system in order to promote health and well-being. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 17 Method Study Design This was a quantitative study using a non-experimental design. Trochim and Donnelly (2007) describes this as a strong and appropriate method of research to uncover new meaning and describe what already exist. Retrospective data was obtained from charts of children evaluated for sensory modulation and auditory processing concerns at a private pediatric clinic in Greenwood, Colorado. The study obtained data from medical charts from January 1, 2010 to July 6, 2018. Before data was extracted from the medical records, the study was approved by the Institutional Review Board (IRB) at Shawnee State University. The Human Research Protection Program at the University of Indianapolis entered into a reliance agreement with Shawnee State University. Participants Potential participants were identified through hands-on examination of medical charts of children age 5 years, 0 months through 11 years, 11 months at the time of testing. The primary researcher (B. W.) determined participant eligibility through a review of their medical record. Inclusion criteria include: Age 5 years, 0 months through 11 years, 11 months. Evaluated at the facility for both sensory modulation and auditory processing concerns as identified by having a sensory modulation assessment (SSP) and an auditory processing assessment (SCAN: C/SCAN-3: C) completed from January 1, 2010 to July 6, 2018. Exclusion criteria included: Children with noted neurological impairment (i.e. stroke, cerebral palsy, traumatic brain injury, and/or peripheral hearing loss). CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 18 Incomplete charts missing composite test scores on the SSP and/or SCAN:C/SCAN-3: C. Any tests (SSP and SCAN: C/SCAN-3: C) in which the evaluator modified the standardized procedures as noted in the evaluation report. Data Collection All data for participants who met the inclusion criteria were collected and entered into a Microsoft Excel spread sheet by the primary researcher. This nonrandomized, purposeful sampling method was used to ensure only charts relevant to answer the research questions were used for this study. The following demographic information was collected from the charts: age, gender, ethnicity, parent education, diagnoses, ear tubes, as well as sub-scores and composite scores of the SSP and sub-scores and composite scores of the SCAN: C/SCAN-3: C. Each participant was assigned a unique study identification number that was used when entering data. Once all data was collected, all participant identifiers were permanently removed. Data was checked for errors, cleaned, and exported into a statistical software program for analysis. Operational definition of variables. This study defined being at-risk for sensory modulation deficits as having a composite score from all seven subtests within the definite difference range (38-141) on the SSP. It also included at least one of the two auditory subtests of the SSP that indicated definite difference. Being at-risk for auditory processing deficits was operationally defined as having a scaled score below 7 on two or more of the four diagnostic subtests (auditory figure-ground, filtered words, competing words-directed ear, competing sentences) on the SCAN: C/ SCAN-3: C. It also included a composite standard score that fell within the disorder range (69 or below). Being at-risk for both SMD and APD were operationally defined as having scores fall within the definite difference range in one or both of the auditory subtests of the SSP and a CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 19 composite score that fell within the definite difference range. It also included having a scaled score below 7 on two or more of the four diagnostic subtests and a composite standard score that fell within the disorder range on the SCAN: C/SCAN-3: C. Instruments Short Sensory Profile. The SSP is a 38-item inventory that uses a 5-point Likert-like scale that ranges from 1 = always to 5 = never and rates the frequency of a childs behavioral response to sensory input (McIntosh et al., 1999). The 38-items are categorized into sensory related groups that include: tactile sensitivity, taste/smell sensitivity, movement sensitivity, under-responsive/seeks sensation, auditory filtering, low energy/weak, and visual/auditory sensitivity. Subtest totals and total composite scores fall within one of three classifications, typical performance, probable difference, or definite difference. Higher scores indicate more typical sensory processing. Composite scores ranges: 155-190 for typical performance, 142-154 for probable difference, and 38-141 for definite difference was used for this study based upon the manual (McIntosh et al, 1999). Subtests for auditory filtering scores range are typical performance range from 23-30, probable difference range from 20-22, and definite difference range from 6-19 and visual/auditory sensitivity scores ranges are: typical performance range from 19-25, probable difference range from 16-18, and definite difference range from 5-15 (McIntosh, 1999). The composite score of all subtests and scores of auditory filtering and visual/auditory sensitivity was used as part of the analysis for this study. The SSP has been found to have good reliability and validity. Cronbachs alpha ranged from .70 - .90 (Auditory Filtering a = .87) for internal consistency. The subtest constructs supported the factor structure of the assessment, ranging from a = .25 to .76, p < .01. Ohl et al. (2012) examined the test-retest of the Sensory Profile, the version which the SSP was obtained CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 20 from. The intraclass correlation coefficient (ICC) results ranged from .50 to .81 (Ohl et al., 2012). It was also determined that 95% of the time the SSP discriminated between children with and those without sensory processing deficits. The SSP is most appropriate to use with children ages 5 years, 0 months through 10 years, 11 months (McIntosh et al., 1999). Children: Tests for Auditory Processing Disorders. The SCAN-3: C is the third edition of the childrens version of Tests for Auditory Processing Skills, which is standardized to screen and diagnose auditory processing deficits. It is used with children ages 5 years 0 months to 12 years 11 months, including both screening subtests and diagnostic subtests (Keith, 2009). The diagnostic portion of the assessment will be used for this study. This includes filter words, auditory figure-ground +8 dB, competing words-directed ear, competing sentences, and a composite score. Scaled scores of 7 or above on each subtest are considered normal and auditory processing weakness is considered when scaled scores below 7 occur on two or more of the subtests (Keith, 2009). Composite scores of 85 or above is considered normal, 70 to 84 is considered borderline, and 69 or below is considered disorder. The test-retest reliability of the SCAN-3: C ranges from ICC = .54 to .77 and internal consistency a = .91 (Keith, 2009). The evidence of test validity of the SCAN-3: C was based upon test content, response processes, internal structure, and special group studies. Results of intercorrelations indicate strong validity with the highest correlation between tests of similar skills (r = .74) and the lowest correlation between tests measure different skills (r = .22) (Keith, 2009). Concurrent and predictive validity were not completed on this assessment or any of the most common standardized assessments used to identify auditory processing disorder (Friberg & McNarmara, 2010). The third edition added ear advantage, which was not used in this study. Both the second edition and third editions were located within the charts of the study. The sub- CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 21 scores and composite scores of both editions used the same standard score of measurement to determine auditory processing results. Only composite scores were used in this study. Procedures Each subject was assigned a unique study identification number that was used when entering data. Once all data was collected and analyzed, all participant identifiers were permanently removed. The primary researcher entered all demographic information and tests scores from medical charts of children that met the inclusion/exclusion criteria. The research assistant pulled medical charts and flagged the data for entry. Scores were double checked by the primary researcher at the time of entry to ensure accuracy when collecting assessment scores prior to entering them into Excel. Methods of ensuring accuracy of entry were by visually double-checking chart with entry, adding subtest scores to ensure they equaled the composite scores, and double-checking sum of scaled scores to the ensure they correlated with the composite score. Any discrepancy was reviewed by the primary investigator and corrected as needed. Data Analysis Data were analyzed using Microsoft Excel and R statistical package (R Core Team, 2013). An alpha level of less than .05 was considered statistically significant. To answer the first question of the number of children at risk for SMD, APD, and/or both disorders, Microsoft Excel was used to determine frequency/percentage. To answer the second research question of determining the relationship between the characteristics of children and being at risk of SMD, APD, and/or both disorders, descriptive statistics were first conducted for the entire sample using Microsoft Excel and R (R Core Team, 2013). Descriptive characteristics included the mean/standard deviation for age, and frequency/percentage for gender (male=0, female=1). CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 22 Diagnoses were identified by parents/guardian on the intake document and were coded as 0 for not having the diagnosis, and 1 as having the diagnosis, which included: Attention Deficit Disorder (no=0, yes=1), Attention Deficit Hyperactive Disorder (no=0, yes=1), Mood Disorder (no=0, yes=1), Autism Spectrum Disorder (no=0, yes=1), Dyslexia (no=0, yes=1), Emotional (no=0, yes=1), Learning Disorder (no=0, yes=1), other (no=0, yes=1). A 0 was coded if the subject had a different race other than white and 1 if the subjects race was white. A 0 was given for each parent of the child if he/she did not have college degree or a 1 if he/she had a college degree. Pressure Equalizer (PE) tubes were coded as 0 meaning no tubes or 1 meaning had tubes inserted. Initially, race was analyzed using Microsoft Excel based on data obtained (white, black, Hispanic, Asian, and other). The percentage was so high for those of the white race that this variable was converted to 0 as non-white and 1 for white. The same procedure was used for the parents level of education. The initial data included high school, 2-year degree, bachelor, masters or doctorate degree. Due to the high percentage of college graduates of a bachelors degree or higher, the variables were change to a 0 indicating no college education or 1 for having a college education. Independent sample t tests were also conducted to answer the second question for subjects at risk of SMD, at risk of APD, and at risk of both across AGE. Independent sample t tests were used in order to test the statistical differences between the means of two groups (Portney & Watkins, 2009). The data met the requirements for this test since AGE is a continuous level of measurement, at risk for each condition is a categorical measurement (yes/no), there was no relationship between the subjects, normal distribution was met, and homogeneity of variances was assumed (Portney & Watkins, 2009). The level of significance CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 23 was set at p < .05 to ensure results are at an acceptable level of probability, thus considered not occurring by chance (University of Connecticut, 2019). Chi Square techniques were also used to determine if there is a true relationship between categorical variables (Statistic Solutions, 2019). The data must meet the requirements of being random, large expected cell counts of 5 or more, and subjects independent of each other (Kent State University, 2019). Field, Miles & Field (2016) suggests a correction to the Pearson formula using Yates Continuity Correction when examining 2x2 continuing tables in order to address inflated Type 1 errors. Using crosstabulation and p < .05, results determine if there is a statistical relationship between the categorical variables (Statistic Solutions, 2019). This study met the stated requirements and examined the relationship between the following categorical predictors; gender (GEN), attention deficit hyperactivity disorder (ADHD), mood (MOOD), autism spectrum disorder (ASD), learning disorder (LD), other (OTH), race (RC), mothers education (MOM), fathers education (DAD), and tubes (TUB) and being at-risk for SMD, APD, and both. Field, Miles, and Field (2016) reported that expected cell frequencies should be greater than five to address low power thus lessening the chance for a Type II error. To further explore the relationship established by Chi Square tests, standard logistic regression can be used to determine a model based upon binominal probability theory (Newsom, 2007). Standard logistic regression is not limited to dichotomous variable as with Chi Square and also not limited to one predictor (Newsom, 2007). Standard logistic regression analysis was performed on subjects that were identified as being at risk for the outcome SMD (Model 1) (0=NO, 1=YES) using ten predictors: age (AGE), gender (GEN), Attention Deficit Disorder (ADD), Attention Deficit Hyperactivity Disorder (ADHD), Mood (MOOD), Autism Spectrum Disorder (ASD), Dyslexia (DYSL), Emotional (EMO), Learning Disorder (LD), and Other CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 24 (OTH). Logistic regression analysis was also performed on subjects that were identified at risk of APD (Model 2) using eight predictors: age (AGE), gender (GEN), Attention Deficit Disorder (ADD), Attention Deficit Hyperactivity Disorder (ADHD), Mood (MOOD), Autism Spectrum Disorder (ASD), Dyslexia (DYSL), and Learning Disorder (LD). Coefficient estimate, odd ratios, 95% confident intervals (CIs), were examined at a .05 significance. Results from Chi Square and logistic regression were able to predict which variable increased the probability of SMD and APD. Due to low cell count for those identified with both conditions (Model 3), results were uninterpretable. Standard logistic regression techniques were followed by backward stepwise regression models. This technique removes nonsignificant characteristics one at a time until only those characteristics that are significant remains (Portney & Watkins, 2009). The final predictor model based on the backwards regression was used to determine cut off points to create adequate sensitivity (true positive) and specificity (true negative) (Pedroli, 2019). A receiver operating characteristic graph (ROC), which has been shown to be a reliable technique for visualizing, organizing, and selecting classifications based on performance was conducted. Swets (1988) found that ROC analysis could be extended for use in visualizing and analyzing behavior of diagnostic systems and for determining accuracy of a test using the area under the curve. Receiver operating characteristic curve findings are interpreted based upon 1.0 being the strongest classification to predict having the behavior and <0.6 being the weakest classification (Pedrol, 2019). Percentage of accurate cases (PAC), sensitivity and specificity, backward elimination models, along with ROC were examined for each model to determine accurate classification and prediction. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS Results Demographic Characteristics Table 1 describes the characteristics of the sample that were analyzed using descriptive statistics. Three hundred and ten cases were determined to meet the inclusion criteria for the study. Results indicated that the majority of the cases were a little older than 7 years old, white males, with parents having a college education. Table 1. Demographic data Descriptive variables Gender (N=310) Females Males Race (N=288) White African American Asian Latin American Hispanic Other Parent Education (N=221) High School Bachelors degree Graduate degree Conditions (N=310) SMD and APD concerns only ADD ADHD Mood Autism Spectrum Disorder Dyslexia Emotional Learning Disability Other Medical Procedure PE Tubes Frequency (Percentage) 78 (25.2%) 232 (74.8%) Mother 11 (5.0%) 125 (56.1%) 86 (38.9%) 237 (82.3.%) 1 (<1%) 8 (2.7%) 9 (3.1%) 12 (4.2%) 22 (7.6%) Father 18 (8.2%) 105 (47.7%) 98 (44.5%) 206 (66.5%) 8 (2.6%) 40 (12.9%) 32 (10.3% 31 (10.0%) 5 (1.6%) 7 (2.3%) 15 (4.8%) 26 (8.4%) 46/16.6% 25 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 26 At-Risk of Sensory Modulation Disorder Three hundred and nine cases had complete SSP scores to analyze. One case was not included due to missing test scores. Using R (R Core Team, 2016), 162 cases (52.4%) were atrisk for SMD based upon the operational definition for this study (definite difference on the SSP composite score and definite difference on at least one of the auditory subtests). Specific to the auditory subtests on the SSP, 147 cases (47.6%) scored definite differences in auditory filtering and a composite score of definite difference. Ninety-two cases (29.8%) scored definite difference in visual/auditory sensitivity and a composite score of definite difference. Seventy-seven cases (24.9%) had both auditory filtering and visual/auditory sensitivity in the definite difference range and a composite score of definite difference. Figure 2 describes types of auditory deficits within SMD. Figure 2. Results of cases at risk of sensory modulation disorder At-Risk for SMD N=309 180 160 140 120 100 80 60 40 20 0 162 147 92 48% Comp/AF 77 30% 25% Comp/VAS Comp/Both 52% SMD Condition Comp (composite score on the Short Sensory Profile), AF (auditory filtering), VAS (visual auditory sensitivity, Both (Auditory filtering and visual auditory sensitivity), SMD Condition (composite score of definite difference and a score of definite difference on at least one auditory subtest) CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 27 The auditory items on the SSP without combining the subtests with the composite scores, indicated deficits (probable and definite differences) occurred most often in the area of SMD auditory filtering (N = 309, 83.2%). This data is noted on the first set of bars (AF Typical, Probable, and Definite) in Figure 3 with additional auditory results included. Figure 3. Auditory subtests of the SSP Auditory Items Only 200 180 160 140 120 100 80 60 40 20 0 184 183 154 103 81 73 52 52 43 63 63 13 AF=Auditory Filter, VAS-Visual Auditory Sensitivity, Both=Auditory Filtering and Visual Auditory Sensitivity, Comp=Composite Score of the SSP At Risk of Auditory Processing Disorder Two hundred and sixty-nine cases had composite scores on the SCAN:C/SCAN-3:C to analyze. Forty-one cases out of the original 310 cases were not included due to missing test scores. Some charts had subtests from one version (SCAN-C) and subtest scores from the newer version of the assessment (SCAN-3: C), thus no composite scores were reported for these 41 cases. Using R (R Core Team, 2016), 16 cases (5.9%) were identified as being at risk of APD based upon the operational definition for this study (composite standard score that fell within the disorder range of 69 or below and a scaled score below 7 on two or more of the four diagnostic subtests). Seventy-seven cases (28.6%) scored within the borderline scale, and one-hundred CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 28 seventy-six cases (65.4%) had composite scores within the normal range. See Figure 2 for details. Figure 4. At risk of Auditory Processing Disorder APD=Auditory Processing Disorder At Risk of Both SMD and APD Conditions Nine cases (N = 268, 3.4%) were at risk of both SMD and APD based upon the operational definition for this study. Thirty-one cases (15.2%) had composite scores on the SSP and the SCAN-C/SCAN-3: C that were in the typical/normal range for both conditions. Eightysix cases (N =268, 32.1%), were at risk for SMD but had normal results for auditory processing. The remaining cases varied in results. Figure 5 describes these outcomes. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 29 Figure 5. At Risk of both SMD and APD SMD=Sensory Modulation Disorder, APD= Auditory Processing Disorder, Def=definite difference, Norm=normal, Border=borderline, Prob=probable. The first term is the results of the SSP and the second term is the results on the SCAN-3: C (SSP/SCAN-3: C). When looking more closely at the SSP auditory processing subtests for those cases that were at risk for both SMD and APD (composite SSP as definite difference and disorder result on the SCAN-C/SCAN-3: C), there were 7 cases that had a subscore of definite difference for auditory filtering. There were 6 cases that had a sub-score of definite difference for visualauditory sensitivity and 4 cases that had sub-scores in the definite difference range for both auditory filtering and visual auditory sensitivities. Figure 6 describes these outcomes. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 30 Figure 6. Auditory subtest results when at risk of both SMD and APD SSP Auditory Subtest Patterns when at risk for Both Conditions 10 8 9 7 6 6 4 4 2 0 AF Comp AP VAS Comp AP Both Comp AP SMD&APD AF: Short Sensory Profile Auditory Filtering subtest, Comp: Short Sensory Profile Composite score, AP: SCAN-C/SCAN:3C composite score Relationship Between Patient Characteristics When Identified as Being at Risk of SMD, APD or Both Conditions Independent sample t tests were conducted for subjects at risk of SMD, at risk of APD, and at risk of both across AGE. Age was not statistically significant between subjects who were at risk of SMD (M= 7.52, SE = 1.84) compared to subjects who were not at risk of SMD (M=7.37, SE=1.83); t (304.4) = - 0.72, p =.47. Likewise, for subjects at risk of APD with those who were not at risk for APD t (17.73) = -1.68, p = 0.11, and at risk for both; t (8.94) = -0.69, p = 0.51. Effect sizes ranged from small to medium (r =. 04 SMD, r = .22 APD, and r = .37 both). The results are presented in Table 2 for all values. Table 2. Age Variables Test df Sig 95 % CI 95% CI Effect Size statistics lower upper SMD -0.72 304.4 .47 -0.56 0.26 .04 APD -1.68 17.73 .11 -1.54 0.17 .37 Both -0.69 8.94 .51 -1.45 0.77 .22 df=degrees of freedom, Sig=p value CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 31 Chi square techniques were used to examine the relationship between the categorical variables; gender (GEN), attention deficit hyperactivity disorder (ADHD), mood (MOOD), autism spectrum disorder (ASD), learning disorder (LD), other (OTH), race (RC), mothers education (MOM), fathers education (DAD), and tubes (TUB) with being at-risk for SMD, APD, and both. Field et al. (2016) reported that expected cell frequencies should be greater than five to address low power, thus lessening the chance for a Type II error. Using test statistic value, the odds ratio was calculated to determine the strength of association of the 10 characteristics and SMD. The results are presented in Table 3 with the variables that emerged with frequencies higher than five for SMD. All variables for APD and both SMD and APD emerged with cell frequencies lower than five, thus were not included. There was a significant relationship between subjects having ADHD and being at risk for SMD, (2 (1) = 4.90, p < 05. The results indicated that the odds of being at risk for SMD were 2.34 times higher (1.10, 5.29) if the subjects had ADHD. Table 3. Chi-square results for SMD with Yates Continuity Correction Variable Test Statistic Sig Exp (B) 95% CI Lower .41 1.1 .61 .88 .75 .73 .50 .13 95% CI Upper 1.2 5.3 3.1 5.0 11. 4.7 1.8 2.5 Gender 1.33 .25 .715 ADHD 4.90 .03* 2.34 Mood 0.42 .52 1.37 ASD 2.59 .11 2.04 LD 1.95 .16 2.60 Other 1.39 .24 1.79 Race 0.00 .99 0.95 Mothers 0.23 .63 0.61 Education Fathers 2.32 .13 0.39 .11 1.2 Education Tubes 0.58 .44 .743 .37 1.5 Sig=p value, Exp (B)= Odds Ratio, CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 32 Prediction of the odds of being at risk for SMD, APD, or both conditions In order to determine if a subjects characteristics predicted the likelihood of being at risk of one or both of these conditions, standard logistic regression analysis with all characteristics entered into the equation at the same time was performed (Portney & Watkins, 2009). Additionally, backward stepwise regression was used by removing nonsignificant characteristics one at a time until only those that were significant remained (Portney & Watkins, 2009). Prediction for SMD (Model 1). First, standard logistic regression analysis was performed on being at risk of SMD (0=NO, 1=YES ) as outcome and ten predictors: age (AGE), gender (GEN), Attention Deficit Disorder (ADD), Attention Deficit Hyperactivity Disorder (ADHD), Mood (MOOD), Autism Spectrum Disorder (ASD), Dyslexia (DYSL), Emotional (EMO), Learning Disorder (LD), and Other (OTH). SMD was identified as Model 1. After deletion of 1 case with missing values, data from n = 309 cases were available for analysis: 147 (47.6%) subjects classified as not being at risk of SMD and 162 (52.4%) subjects classified as being at risk of SMD. Analysis was performed using R (R Core Team, 2015). Race, mother and fathers education, and tubes were eliminated due to a high percentage of missing data and limited variation. A test of the full model with all ten predictors against a constant-only model was not statistically reliable, 2 (11, N = 309) = 13.0, p = .29, indicating that the set of predictors did not reliably distinguish between those who were at risk of SMD and those who were not at risk of SMD. The variance in risk of SMD was small using McFaddens Pseudo R Square = 0.034, df = 11. Prediction success (using 0.5 as the threshold) was unimpressive with 178 of 309 cases (57.6%) accurately classified and predicted correctly with sensitivity and specificity values of 0.38 and 0.79, respectively. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 33 Table 4 shows regression coefficients, Wald statistics, odds ratios, and 95% confidence intervals for odds ratios for each of the ten predictors. According to the Wald criterion, only ADHD reliably predicted being at risk of SMD, z = 2.24, p < .05. A model run with ADHD omitted was not reliability different from a constant only model; however, this model was reliably different from the full model, 2 (1, N = 309) = 5.48, p < .05, which confirmed that ADHD is the only reliable predictor of being at risk for SMD among the set of predictor variables. The odds ratio of 2.64 showed significant change in the likelihood of being at risk for SMD based on being diagnosed with ADHD. Variance Inflation Factors (VIF) values ranged from 1.04 (Other) to 1.28 (ADHD), indicating that multicollinearity was not a problem. Examination of the significance levels of the additional predictors created by examining the interaction between each quantitative predictor (AGE) and the log of itself (Hosmer & Lemeshow, 1989) indicated that a linear relationship between the predictor variables and the being at risk of SMD status may be assumed. Table 4. Logistic Regression for At-Risk for Sensory Modulation Disorder (Model 1) Variables B Intercept Age Gender ADD ADHD Mood ASD Dyslexia Emotional -0.11 0.01 -0.26 -0.01 0.97 -0.13 0.50 -1.66 -0.10 LD 0.79 SE .49 .07 .27 .80 .43 .43 .42 .07 .84 .67 Wald 0.22 0.17 0.94 0.02 2.24 -0.31 1.17 -1.55 -0.12 1.18 Exp (B) 1.01 0.77 0.99 2.64 0.87 1.64 0.19 0.90 2.21 Sig .82 .87 .35 .99 .03* .76 .24 .12 .91 .24 95% CI Lower -0.18 -0.22 -0.74 -1.16 -0.26 -1.38 -0.25 -3.75 -0.18 -1.12 95% CI Upper 3.34 0.13 0.68 2.72 2.17 0.63 2.08 1.02 5.16 2.02 B=Regression Coefficient, SE=Standard Error, Exp (B)= Odds Ratio, Sig=p value CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 34 The standard logistic regression analysis was followed up with a backward elimination logistic regression analysis for SMD in order to identify a reduced model that best explains the data (Hocking, 1976). Using this analysis reduces the risk of multicollinearity. Beginning with the full set of predictor variables, after 4 Fisher Scoring iterations, a statistically reliable reduced model emerged, 2 (4, N = 309) = 9.6, p < .05 with three predictors: ADHD, Dyslexia, and LD. The variance accounted for being at risk for SMD remained unimpressive with McFaddens rho = 0.02, df = 4. Prediction success decreased slightly to 54.0% (167 of 309 cases). Sensitivity and specificity values changed slightly to 0.20 and 0.91, respectively. Table 5 displays the regression coefficients, Wald statistics, odds ratios, and 95% confidence intervals for odds ratios for the remaining three predictors. According to the Wald criterion, only ADHD emerged significantly (z = 2.48, p < .05). The odds ratio of 2.75 indicated that when controlling for dyslexia and LD, the odds of subjects being at risk for SMD was 2.75 times higher for those with ADHD. Table 5. Backward Stepwise Regression for At-Risk for Sensory Modulation Disorder (Model 1) Variables B SE Wald Exp B Sig 95% CI 95% CI Lower Higher Intercept -0.04 0.12 -0.30 0.96 .76 0.76 1.23 ADHD 1.01 0.41 2.48 2.75 .01* 1.28 6.45 Dyslexia -1.78 1.04 -1.71 0.17 .08 0.02 1.26 LD 0.96 0.64 1.50 2.60 .13 0.81 10.43 B=Regression Coefficient, SE=Standard Error, Exp (B)= Odds Ratio, Sig=p value CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level The three-predictor model was used to determine cut off points to create adequate sensitivity and specificity. Sensitivity is the probability that an individual that has a condition will test positive for the condition (Davidson, 2002). This is calculated by the true positive divided by the total calculation of true positive and false negative. Specificity is the probability that an individual that does not have a condition will test negative for the condition. This is calculated by the true negative divided by the total calculation of true negative and false CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 35 positives (Davidson, 2002). A receiver operating characteristic graph (ROC), which has been shown to be a reliable technique for visualizing, organizing, and selecting classifications based on performance, is presented in Graph 1. Swets (1988) found that ROC analysis could be extended for use in visualizing and analyzing behavior of diagnostic systems and for determining accuracy of a test using the area under the curve. The ROC curve is the plot of true positive rate against false positive rate in order to determine possible cut points for having a condition, also known as a probability curve (Narkhede, 2018; Portney & Watkins, 2009). The Area Under the Curve (AUC) describes the degree the model distinguishes between classes (Narkhede, 2018). For the set of predictors, the area under the curve was found to be 0.567, which indicated a very poor accuracy classification for this at-risk behavior (Tape, 2015). Results are based upon 1.0 being the strongest classification to predict having the behavior and <0.6 being the weakest classification. Results from logistic regression, when taken in isolation of other statistics, should be generalized with caution when classifications are poor. Graph 1. ROC curve for intent CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 36 In order to lessen error of misclassification of presence-absence of a condition, the calculation of threshold probability supports the modeling technique of prediction and improves the confidence of its accuracy (Jimenez-Valverde & Lobo, 2007). Minimized difference threshold (MDT) is calculated by the difference between sensitivity and specificity that is obtained from the AUC method (Goberville, Beaugrand, Hautekeete, Piquot, & Luczak, 2015). Using R to calculate MDT, it was found that 0.537 minimizes the absolute difference between sensitivity and specificity. The values of the sensitivity and specificity at 0.537 were 0.912 and 0.204, respectively. This method of prediction is considered better due to removing false positive presence (Goberville et al., 2015). Graph 2 shows the plot of model sensitivity and specificity for various cut points. Graph 2. Plot of model sensitivity and specificity for various cutoffs Prediction for APD (Model 2). The next condition that was analyzed was being at risk for APD, identified as Model 2. Logistic regression was completed to predict the odds of being at risk of APD based upon the subjects characteristics. Standard logistic regression analysis was performed on being at risk for APD as the outcome with eight predictor variables: age (AGE), gender (GEN), Attention Deficit CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 37 Disorder (ADD), Attention Deficit Hyperactivity Disorder (ADHD), Mood (MOOD), Autism Spectrum Disorder (ASD), Dyslexia (DYSL), and Learning Disorder (LD). After deletion of 40 cases with missing values for being at risk for APD, data from n = 269 cases were available for analysis: 253 (94.1%) subjects classified as not being at risk for APD and 16 (5.9%) subjects classified as being at risk for APD. Analysis was performed using R (R Core Team, 2015). Race, mother and fathers education, and tubes were eliminated due to a high percentage of missing data and limited variation. Emotional Disorder and Other were eliminated due to an extremely high standard error. A test of the full model with all eight predictors against a constant-only model was statistically reliable, 2 (9, N = 269) = 95.2, p = .001 indicated that the set of predictors reliably distinguish between those who were at risk for APD and those who were not at risk for APD. The variance in risk of APD accounted for was moderate with McFaddens Pseudo R Squared = 0.111, df = 9. Prediction success (using 0.5 as the threshold) was impressive with 253 of 269 cases (94.1%) accurately classified and predicted correctly with sensitivity and specificity values of 0.00 and 1.00 respectively. Table 6 shows regression coefficients, Wald statistics, odds ratios, and 95% confidence intervals for odds ratios for each of the eight predictors. According to the Wald criterion, only ASD reliably predicted being at risk of APD, z = 3.05, p < .01. A model run with ASD omitted was not reliability different from a constant only model; however, this model was reliably different from the full model, 2 (1, N =269) = 8.29, p < .01 which confirmed that ASD is the only reliable predictor of being at risk for APD among the set of predictor variables. The odds ratio of 8.21 showed significant change in the likelihood of being at risk for APD based on being diagnosed with ASD. Variance Inflation Factors (VIF) values ranged from 1.06 (AGE) to 1.91 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 38 (DYSL), indicating that multicollinearity was not a problem. Examination of the significance levels of the additional predictors created by examining the interaction between each quantitative predictor (AGE) and the log of itself (Hosmer & Lemeshow, 1989) indicated that a linear relationship between the predictor variables and the being at risk of APD status may be assumed. Table 6: Logistic Regression for At-Risk for Auditory Processing Disorder (Model 2) Variables Intercept Age Gender ADD ADHD Mood ASD Dyslexia LD B -5.11 0.24 0.61 1.51 -1.10 -0.46 2.11 2.58 -0.90 SE Wald 1.28 0.16 0.64 1.23 1.10 0.92 0.69 1.67 1.30 -3.99 1.58 0.96 1.22 -1.00 -0.50 3.05 1.54 -0.469 Odds Ratio 0.006 1.269 1.847 4.499 0.334 0.630 8.214 13.174 0.406 Sig 95% CI Lower .00*** 0.000 .11 0.944 .34 0.494 .22 0.200 .32 0.017 .62 0.075 .00** 2.041 .12 0.366 .49 0.016 95% CI Upper 0.065 1.718 6.476 39.291 1.967 3.144 32.264 489.657 3.520 B=Regression Coefficient, SE=Standard Error, Exp (B)= Odds Ratio, Sig=p value CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level The standard logistic regression analysis was followed up with a backward elimination logistic regression analysis. Beginning with the full set of predictor variables, after 6 Fisher Scoring iterations, a statistically reliable reduced model emerged, 2 (3, N = 269) = 9102.2 p < .001 with two predictors: AGE and ASD. The variance accounted for being at risk for APD remains unimpressive with McFaddens rho = 0.007, df = 3. Prediction success sensitivity and specificity values remained unchanged. Table 7 displays the regression coefficients, Wald statistics, odds ratios, and 95% confidence intervals for odds ratios for the remaining two predictors. According to the Wald criterion, only ASD emerged significant (z = 2.76, p < .01). The odds ratio of 5.12 indicated that when controlling for age, the odds of subjects being at risk for APD is 5.12 times higher for subjects with ASD. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 39 Table 7. Backward Stepwise Regression for At-Risk for APD (Model 2) Variables B SE Wald Exp (B) Sig 95% CI 95% CI Lower Higher Intercept -4.74 1.25 -0.38 0.009 .00*** 0.001 0.089 AGE 0.21 0.15 1.46 1.237 .14 0.929 1.656 ASD 1.63 0.06 2.76 5.123 .00** 1.488 15.808 B=Regression Coefficient, SE=Standard Error, Exp (B)= Odds Ratio, Sig=p value CI=Confident Interval, * significant at .05 level; ** significant at .01 level; *** significant at .001 level The two-predictor model was statistically reliable and used to determine cut points to create adequate sensitivity and specificity. A receiver operating characteristic graph (ROC), which has been shown to be a reliable technique for visualizing, organizing, and selecting classifications based on performance, is presented in Graph 3. Swets (1988) found that ROC analysis could be extended for use in visualizing and analyzing behavior of diagnostic systems and for determining accuracy of a test using the area under the curve. For the set of predictors, the area under the curve was found to be .721, which indicated a fair accuracy classification for this diagnostic (Tape, 2015). Graph 3. ROC curve for intent CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 40 Graph 4 shows a plot of model sensitivity and specificity for various cutoffs. Using R and the minimized difference threshold (MDT) (Jimnez-Valverde & Lobo, 2007), it was found that 0.054 minimizes the absolute difference between sensitivity and specificity. The values of the sensitivity and specificity at 0.054 were 0.656 and 0.688, respectively. Graph 4. Plot of model sensitivity and specificity for various cut points Prediction for SMD and APD (Both) (Model 3). Standard logistic regression was run for the final condition of being at risk of SMD and APD (both), identified as Model 3. Results indicated the statistics were unstable due to low cell count. Summary of Results This descriptive study reviewed 310 cases that met the inclusion/exclusion criteria in order to answer three objectives. The first objective addressed the number of children at-risk of sensory modulation deficits, at-risk of auditory processing deficit, or at-risk of both conditions. One case was removed due to missing SSP scores, with results of 162 cases (52.4%) being atrisk of SMD based upon the operational definition for this study (definite difference on the SSP CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 41 composite score and definite difference on at least one of the auditory subtests). Forty-one cases were missing data leaving 269 cases with composite scores on the SCAN:C/SCAN-3: C. Sixteen cases (5.9%) were identified as being at risk of APD based upon the operational definition for this study (composite standard score that fell within the disorder range of 69 or below and a scaled score below 7 on two or more of the four diagnostic subtests). Lastly, a total of 268 cases had scores for both assessments, with 9 cases (3.4%) being at risk of both SMD and APD based upon the operational definition for this study. The second objective for this study was to determine if there was a relationship between patient characteristics and being at risk of one or both of these conditions. The third objective was based upon these results in order to determine predictors of each condition. Results indicated that age was not statistically significant thus not related to being at risk of SMD, APD, or both of these conditions. Of all the characteristics examined in this study, results indicated that ADHD (p=.03) was related and significantly predictive of SMD. The odds ratio of 2.75 indicated that when controlling for dyslexia and LD, the odds of subjects being at risk for SMD was 2.75 times higher for those with ADHD. Results indicated that ASD (p=.00) was related and significantly predictive of APD. The odds ratio of 5.12 indicated that when controlling for age, the odds of subjects being at risk for APD was 5.12 times higher for subjects with ASD. Discussion The purpose of this study was three-fold. The first purpose was to determine the number of children at-risk of sensory modulation deficits, at-risk of auditory processing deficit, or at-risk for both. Secondly, to determine if there was a relationship between patient characteristics and being at risk of one or both of these conditions. Since warranted, predictors were determined for each condition. Results from this study may be used to support a clearer understanding of the CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 42 relationship between these two pediatric conditions and strengthen the ability to screen and refer these children appropriately. Results also have the potential to support therapists to design and implement treatments that are specific to the characteristics of identified deficits and support future direction of research. The clinical sample used in this study was based upon children referred to a pediatric clinic that specializes in sensory processing challenges in children and adults. Prevalence of children identified at risk of SMD tend to be significantly higher in this setting due to children being referred exhibiting behaviors indicating the need for this type of assessment. Results indicated that 59% (N=183) of the children referred for SPD scored in the definite difference range for being at risk of SMD with an additional 20% (N=63) that had probable difference. The additional 20% cases that had probable difference on the SSP, identified a group of children that would typically fall within Tier 2 of the Response to Intervention school model (Positive Behavioral Interventions & Supports [PBIS], 2019). Tier 2 includes educational supports for children that are not successful in the traditional educational approach (Tier 1) but deficits are not to the degree that require an Individual Educational Plan (IEP), which is Tier 3 (PBIS, 2019). Results of this study indicate that educational staff needs to understand characteristics of SMD that would benefit from supports within the Tier 2 system when screening results indicate probable difference in SMD that include deficits in the auditory subtests. These results also indicated a high rate of appropriate referrals. The referral source was not examined in this study, but results may indicate a better understanding of behaviors that need evaluated by occupational therapists that specialize in sensory modulation disorder. The clinical sample in this study was unique compared to other published studies because it looked specifically at the prevalence of SMD that involved auditory deficits. Fifty-two percent CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 43 of the cases that had definite difference on the SSP involved deficits in the auditory system. Even though auditory concerns were significant enough to warrant screening for auditory processing disorder, this was not the reason for auditory deficits for the majority of children. A small percentage of children were at risk of APD, which was similar to the rates reported by AAA (2018). Co-morbidity of both SMD and APD was also very small indicating the uniqueness of auditory deficits specific to SMD. No other published studies were identified that compared the prevalence of co-morbidity of auditory impairment in SMD and APD. While the current study indicated that co-morbidity is small, additional research is needed. Results indicate that OTs need to ensure they evaluate for auditory issues within SMD and specifically address this sensory system in treatment when warranted. More OTs need to specialize in the auditory system within SMD to determine best practice. The development and future publication of the SP-3D (Schoen, Miller, & Sullivan, 2016) that identifies specific behaviors of auditory SUR, SOR, and SC will greatly contribute to identification of the specific area(s) of auditory deficits when evaluating children for SMD. The SSP does not measure each type of sensory modulation deficits in the area of auditory sensation, thus is limited in interpretation. Auditory filtering is a mixture of responses and the SSP combines visual and auditory over responsivity within the same subtest. The lack of published comprehensive tools for each type of SMD hinders identifying and treating the specific auditory SMD area. It also limits appropriate measurements for pre-post research studies to determine efficacy in treatment for SMD within the auditory domain. It is unknown if there is a relationship between auditory SOR and auditory filtering within SMD. Eighty-one cases of the 309 had both SOR and AF deficits but as previously stated, the design of the SSP overlaps SOR, SUR, SC within the AF subtest and auditory SOR is combined with visual SOR. Only 2 of the 5 items of the subtest of visual/auditory sensitivity CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 44 address auditory SOR. When looking specifically at the items in the auditory filtering subtest of the SSP, behaviors such as distracted by a lot of noise, cant work with background noise, and doesnt respond when name is called but you know the childs hearing is OK, raise questions if SOR influences AF and how AF is different from auditory processing that is tested by audiologists. We do know that deficits in AF and SOR have the potential to negatively influence a childs success in academics due to the complexity of the auditory environment in the school setting (Bradlow, Kraus, & Hayes, 2003; Shield & Dockrell, 2008). School is an environment that children spend the majority of their time, focusing on academics, social participation, and activities of daily living (eating & toileting). In the school setting, children are screened for peripheral hearing loss by nursing, but not screened for auditory deficits in the areas of APD or SMD (Ohio Department of Health, 2015). A better understanding of the relationship between auditory filtering and auditory SOR would improve the identification and provision of a supportive learning environment for this population. Some schools are using Frequency Modulation (FM) systems within classrooms to encourage auditory attention (Reynolds, Kuhaneck & Pfeiffer, 2016). The FM system allows the teacher to speak into a microphone, which transmits their voice throughout the classroom through speakers (Kreisman & Crandell, 2002). The objective of this system is to improve the signal-to-noise ratio, thus supporting the teachers voice to be heard over background noise (Kriesman & Crandell, 2002). There are limitations in many of these studies, but overall a general body of evidence is growing indicating that FM systems support children with learning difficulties (Reynolds et al., 2016; Schafer, Mathews, Mehta, Hill, Munoz, Bishop & Moloney, 2013; Updike, 2006). The challenge of depending upon FM systems for school success is that they are not mandated as a universal design for public schools and the group of children with CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 45 SMD in the area of auditory deficits has not been directly studied to determine its efficacy with this population (Reynolds et al., 2016). In addition, school systems often use noise cancellation headphones to help children cope when they have auditory SOR. This is a compensatory technique that does not target remediating the cause of auditory SOR. In fact, the literature indicates that noise cancellation headphones may cause an increase in auditory SOR due to the gating system being activated less (Dozier, 2016; Frick, 2016). All treatment designed to address auditory SOR need to be studied for efficacy. For the sample population in this study, children referred for testing due to behaviors of SMD and APD were primarily boys. Prior research has indicated boys have a higher incidence of both SMD and APD (Chermak & Musick, 1997; Schoen et al., 2018). This study further identified a higher referral rate of boys being at risk of processing difficulties within the auditory system. Identifying deficits and providing intervention for children as early as possible has the potential to lessen the severity of impairment into adulthood (Rice et al., 2014). The mean age of children referred and identified at-risk for the stated conditions in this study was at the secondgrade level of education or 7-8 years old. The inclusion criteria of the study were based upon the age range of the SCAN-3: C, which starts at age 5. Shapiro (2016) reported that children are typically referred for APD testing at age 7 or 8 years old but believes this should occur at a much younger age. Typically, children are identified with SMD during the early years of school when the demands of a multi-sensory environment impede school function. This study was able to determine the mean age of children referred for SMD in a clinical setting based upon the inclusion criteria of 5-11 years old, ages used by the SCAN-3: C. In the general population, the mean age may be younger due to availability and widely used tools that measure SMD in the CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 46 infant and toddler age group (Dunn, 2008). It is important to acknowledge that Ahn et al. (2004) identified 5.3% of the kindergarten population met the criteria for SMD using the SSP as a screening tool. The current challenge is providing adequate screening for all auditory deficits for children at the birth to 5-year-old period of development, thus increasing family and teacher education and intervention for children prior to the childhood demands of school. In 2009, Part C (Early Intervention program) served 2.67% of the general population of birth to three-year-old children (National Early Childhood Technical Assistance Center, 2011). Based upon stated data, up to 13% of this age group met the criteria for service, yet those families did not choose to participate in the program (National Early Childhood Technical Assistance Center, 2011). Screening tools are available to capture infants, toddlers, and preschoolers that are at risk of SMD but tools are lacking to adequately screen this age group for being at risk of APD. Since the SCAN-3: C starts at age 5 and is a commonly used screening tool among practitioners, it becomes a challenge for early identification of the at-risk population of APD. Identifying at-risk behavior of APD during the preschool years has shown promise using neural coding of speech in noise and phonology but that is not common practice (White-Schwoch et al., 2015). Affordable and reliable assessments needs developed to capture preschoolers at-risk of APD and improved multi-disciplinary team approach that have an understanding of auditory deficits in the younger population. The second purpose of this study was to determine if there was a relationship between patient characteristics and being at risk of one or both of these conditions. Results indicated a relationship between being at risk of SMD and ADHD at a significant level. This was not surprising based upon the growing body of evidence indicating co-morbidity among SOR and ADHD (Lane & Reynolds, 2019). In fact, one study used the SSP, controlling for overlapping CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 47 characteristics of AF and sensory seeking among SMD and ADHD, with results still indicating co-morbidity. Yet, ADHD and SMD did occur alone, as distinct conditions independent of each other (Miller et al., 2012). The current study was unable to control for behaviors similar to attention deficits and may have inflated the result. However, it is important to note that the outcomes in the study by Miller et al. (2012) still demonstrated AF deficits in cases with SMD, ADHD, and both conditions compared to typically developing children. The relationship between APD and the identified characteristics were examined indicating a relationship between autism spectrum disorder (ASD) and APD. Co-morbidity of APD and ASD has been reported throughout the literature (Brandwein et al., 2015; Danesh & Kaf, 2012; Demopoulos et al., 2015; Denman, Banajee, & Hurley, 2015; DePape, Hall, Tillmann, & Trainor, 2012; Kozou, Azouz, Abdou, & Shaltout, 2018; Linke, Keehn, Pueschel, Fishman, & Muller, 2018; Lortie et al., 2017; Ocak, Eshraghi, Danesh, Mittal, & Eshraghi, 2018, and Otto-Meyer, Krizman, White-Schwoch, & Kraus, 2018.) When reflecting on the diagnostic criteria of ASD including language delay and social pragmatic deficits, it is not surprising that results indicated a relationship between APD and ASD. The results of this study did not identify a relationship of being at risk of SMD with ASD, which is surprising. Literature has repeatedly identified auditory sensitivity in individuals with autism (Bhatara., Quintin, Fombonne, & Levitin, 2013; Danesh & Kaf, 2012; Linke et al., 2018; Matsuzaki et al., 2017). The nature of the assessment (SOR in both auditory and visual SOS combined and only 2 of 5 items addressing SOR) used in this study may not have been able to determine the relationship of SMD and ASD. Additional studies using assessment tools measuring auditory SOR in individuals with ASD needs completed. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 48 The current study expands the body of knowledge indicating that the odds of children with SMD having ADHD is 2.72 times higher than being at risk of one condition alone. Knowing that the auditory items of the SSP overlap with the diagnostic criteria of ADHD may have inflated the odds of having both conditions, yet based upon the results of the study by Miller el al., (2012), auditory filtering scores indicated clinical impairment for both SMD and ADHD using an inclusion criteria that controlled for in-attention. Miller et al., (2012) also suggested that children referred for ADHD should be screened for SMD. The screening, evaluation, and treatment of children at risk of SMD and/or ADHD should include auditory specific testing and appropriate treatment based upon the emerging evidence. The probability of having APD and ASD was at a very high rate (5.12) in this study. Individuals with ASD have significant deficits in communication and social pragmatics, which are based upon the DSM-5 diagnostic criteria (Center for Disease Control and Prevention, 2019). Children with ASD should always be screened for APD and treated accordingly. Evaluating this population can be challenging but having a complete understanding of all areas of APD is important for appropriate treatment. Limitations Data were collected at one clinic that specialized in SPD, lacking diversity as seen in the general population. Results can be applied to this specific clinic but needs repeated using a more diverse population in order to generalize the results. Another limitation is how the data were collected. A retrospective study uses data obtained from several therapists and over a period of several years. There was no ability to determine the accuracy of administration of the assessments, the environment during administration, and how the parent and child felt at the time of being evaluated. It is noted that the clinic has a reputation of high-quality therapists and CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 49 services. Lastly, there were limitations in the assessments used. The Short Sensory Profile is a parent report inventory thus lacked behavior-based administration in order to determine accuracy of responses. The auditory items on the SSP did not separate SOR, SUR, SC and combined vision and auditory SOR on one subtest, which limited the ability to identify the auditory subtest according to recent publications of SMD (Miller et al., 2007). Clinical Relevance and Future Research This study provided data comparing characteristics of children that were referred for an evaluation based upon sensory deficits that included auditory deficits with no peripheral hearing loss. Results indicated that the majority of children with auditory difficulties were at risk of SMD involving auditory deficits versus APD. Clinically, auditory deficits are complex, and a multidisciplinary team needs to be involved that specializes in this area of practice. Health care professionals need a better understanding of symptoms of auditory deficits in order to make appropriate referrals. There also needs to more research exploring the relationship of AF and SOR in order to treat appropriately. This study determined being at risk of SMD using the SSP that included two criteria, composite score of definite difference and definite difference on at least one of the auditory related subtests. Additional studies are needed that separate the three auditory areas, that include auditory under-responsivity, auditory over-responsivity, and auditory sensory craving in order to guide best method of evaluating and treating SMD with specific auditory involvement. This is especially needed in the area of over-responsivity. There is a growing body of literature among various professions that are exploring auditory over-responsivity (Danesh & Kaf, 2012; OttoMeyer et al., 2018). Occupational therapists evaluate and treat functional impairment, such as coping strategies to attend events that are noisy or sound based treatment targeting the ability to CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 50 lessen the sensitivity to specific tones when flushing the commode or drying hands in the bathroom. Further research is needed to understand auditory SOR and determine the effectiveness of treatment used by occupational therapists and their ability to improve functional outcomes. Based upon the results of this study, children with a diagnosis ADHD should be evaluated for SMD and children with ASD should be evaluated for APD due to being at a greater risk of these conditions. Pediatricians needs to understand the probability of co-morbidity of these conditions in order to make appropriate referrals. Therapists need to understand the difference in these conditions in order to treat appropriately. Additional studies are needed beyond screening for SMD and APD by comparing the characteristics and neurology of children using diagnostic testing. There continues to be a gap in the literature that identifies the neurological difference between SMD and APD. Preliminary research has explored non-classical auditory pathways (Moller, Kern, & Grannemann, 2005); vagal modulation to auditory stimuli (Cruz, Ferreira-Santos, Oliveira-Silva, Ribeeriro, Goncalves, & Sampaio, 2017); P300 (Ubiali, Sanfins, Borges & Colella-Santos, 2016), and electro- and magnetoencephalography methods of the auditory pathway (Orekhova et al., 2012), yet there is so much that is unknown when considering the underlying neurology of these two disorders. In conclusion, there is a growing body of literature that has explored auditory pathways, auditory behaviors indicating dysfunction, how symptoms overlap among various diagnoses, yet some symptoms are unique for some conditions. This study contributes to the growing body of literature in several ways. First, auditory deficits in children at risk of SMD is not the same as the auditory deficits in children at risk of APD. Having appropriate referrals made by pediatricians CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 51 and therapists knowledgeable of auditory impairment in SMD can guide appropriate evaluation and treatment process for these children. Second, children at risk of ADHD should be evaluated for SMD and vice versa. Families, physicians, and other professionals that are in the position to make referrals or determine what needs to be evaluated should keep this in mind. The same is true for children at risk for ASD. They should be assessed for APD and vice versa. When specific symptoms are identified, specific treatment can be formulated, increasing the likelihood of an improved quality of life for the children and their families. Lastly, occupational therapists are taught the inter-relationship of the individual, the occupations or activities that the individual engages in, and how context and environment influences performance (AOTA, 2014). Our unique holistic approach can provide valuable information to the team when working with individuals with auditory dysfunction that impairs life. Being consumers and participants of research in the area of auditory deficits can insure our presence on the team. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 52 References Aazh, H., Moore, B. C., & Prasher, D. (2011). Providing support to school children with hyperacusis. British Journal of School Nursing, 6(4), 174-178. doi:10.12968/bjsn.2011.6.4.174 Ahn, R. R., Miller, L. J., Milberger, S., & McIntosh, D. N. (2004). Prevalence of parents' perceptions of sensory processing disorders among kindergarten children. The American Journal of Occupational Therapy, 58(3), 287-293. doi:10.5014/ajot.58.3.287 American Academy of Audiology (2018). Central auditory processing disorder: Practice guidelines. Retrieved from https://www.audiology.org/publications-resources/documentlibrary/central-auditory-processing-disorder. American Occupational Therapy Association (2014). Occupational therapy practice framework: Domain & process (3rd ed.). American Journal of Occupational Therapy, 68(Suppl. 1), S1 S48. http://dx.doi.org/10.5014/ajot.2014.682006 American Speech-Language-Hearing Association. (2005). (Central) auditory processing disorder [Technical Report]. Retrieved from https://www.asha.org/policy/TR200500043/. Ausderau K. K., & Baranek, G. T. (2013). Sensory experiences questionnaire. In F. R. Volkmar (Eds.) Encyclopedia of autism spectrum disorders. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-1698-3 Barrasso-Catanzaro, C. & Eslinger, P. J. (2016). Neurobiological bases of executive function and social-emotional development: Typical and atypical brain changes. Family Relations, 65, 108-119. doi:10.1111/fare.12175 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 53 Bellis, T. J. (nd). Understanding auditory processing disorders in children. Retrieved from https://www.asha.org/public/hearing/Understanding-Auditory-Processing-Disorders-inChildren/ Ben-Sasson, A., Carter, A. S., & Briggs-Gowan, M. J. (2009). Sensory over-responsivity in elementary school: Prevalence and social-emotional correlates. Journal of Abnormal Child Psychology, 37, 705-716. doi:10.1007/s10802-008-9295-8 Bhatara, A., Quintin, E.-M., Fombonne, E., & Levitin, D. J. (2013). Early sensitivity to sound and musical preferences and enjoyment in adolescents with autism spectrum disorders. Psychomusicology: Music, Mind, and Brain, 23(2), 100108. https://doiorg.proxy01.shawnee.edu/10.1037/a0033754.supp (Supplemental) Bradlow, A. R, Kraus, N., & Hayes, E. (2003). Speaking clearly for children with learning disabilities: Sentence perception in noise. Journal of Speech, Language, and Hearing Research, 46(1), 80-97. Retrieved from http://search.escohost.com/login.aspx?direct=true&db=izh&AN=106786582&site=edslive&scope=site Brandwein, A., Foxe, J., Butler, J., Frey, H.-P., Bates, J., Shulman, L., & Molholm, S. (2015). Neurophysiological Indices of Atypical Auditory Processing and Multisensory Integration are Associated with Symptom Severity in Autism. Journal of Autism & Developmental Disorders, 45(1), 230244. https://doiorg.proxy01.shawnee.edu/10.1007/s10803-014-2212-9 Center for Disease Prevention and Control (2019). Developmental screening. Retrieved from https://www.cdc.gov/ncbddd/childdevelopment/screening.html. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 54 Chermak, G. D. & Musik, F. (1997). Central auditory processing disorders: New perspectives. San Diego, CA: Singular Publishing Group. Chermak, G. D., Tucker, E., & Seikel, J. A. (2002). Behavioral characteristics of auditory processing disorder and attention-deficit hyperactivity disorder: Predominantly inattentive type. Journal of American Academy of Audiology, 13, 332-338. Child Mind Institute (2018). Quick facts on auditory processing disorder. Retrieved from https://childmind.org/article/quick-facts-on-auditory-processing-disorder/. Cosbey, J., Johnston, S. S., & Dunn, L. (2010). Sensory processing disorders and social participation. American Journal of Occupational Therapy, 64(3), 462-473. doi:10.504/ajot2010.09076 Crozier, S. C., Goodson, J. Z., Mackay, M. L., Synnes, A. R., Grunau, R. E., Miller, S. P., & Zwicker, J. G. (2016). Sensory processing patterns in children born very preterm. American Journal of Occupational Therapy, 70(1), 1-7. doi:10.5014/ajot.2016.018747 Cruz, S., Ferreira, S. F., Oliveira, S. P., Ribeiro, E., Gonalves, ., & Sampaio, A. (2018). Vagal modulation of 1monthold infants to auditory stimuli is associated with selfregulatory behavior. Social Development, 27(2), 322334. https://doiorg.proxy01.shawnee.edu/10.1111/sode.12270 Danesh, A. A., & Kaf, W. A. (2012). DPOAEs and contralateral acoustic stimulation and their link to sound hypersensitivity in children with autism. International Journal of Audiology, 51(4), 345352. https://doiorg.proxy01.shawnee.edu/10.3109/14992027.2011.626202 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 55 Davidson, M. (2002). The interpretation of diagnostic tests: A primer for physiotherapists. Australian Journal of Physiotherapy, 48, 227-232. Demopoulos, C., Hopkins, J., Kopald, B. E., Paulson, K., Doyle, L., Andrews, W. E., & Lewine, J. D. (2015). Deficits in auditory processing contribute to impairments in vocal affect recognition in autism spectrum disorders: A MEG study. Neuropsychology, 29(6), 895 908. https://doi-org.proxy01.shawnee.edu/10.1037/neu0000209 Denman, I., Banajee, M., & Hurley, A. (2015). Dichotic listening training in children with autism spectrum disorder: A single subject design. International Journal of Audiology, 54(12), 991996. https://doiorg.proxy01.shawnee.edu/10.3109/14992027.2015.1070308 DePape, A.-M. R., Hall, G. B. C., Tillmann, B., Trainor, L. J., & Bremner, A. (2012). Auditory Processing in High-Functioning Adolescents with Autism Spectrum Disorder. PLoS ONE, 7(9), 117. https://doi-org.proxy01.shawnee.edu/10.1371/journal.pone.0044084 de Wit, E., van Dijk, P., Hanekamp, S., Visser-Bochane, M. I., Steenbergen, B., van der Schans, C.P., & Luinge, M. R. (2017). Same or different: The overlap between children with auditory processing disorder and children with other developmental disorders: A systematic review. Ear & Hearing, 39(1), 1-19. doi:10.1097/AUD.0000000000000479 Dozier, T. (2016). Earplugs and noise cancelling headphones. Retrieved from https://misophoniainstitute.org/earplugs-and-noise-cancelling-headphones/ Dunn, W. (1999). Sensory Profile [manual]. San Antonio, TX: Psychological Corporation. Dunn, W. (2008). Infant/Toddler Sensory Profile [manual]. San Antonio, TX: Psychological Corporation. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 56 Educational Management Solutions (2017). Six benefits of interprofessional collaboration. Retrieved from https://www.simulationiq.com/blog/content/6-benefits-interprofessionalcollaboration. Ferre, J. M. (2015). Auditory dysfunction beyond the 8th nerve: Understanding central auditory processing disorders. Perspectives on Hearing and Hearing Disorders: Research and Diagnostics, 19, 4-11. doi:10.1044/hhd19.1.4 Field, A., Miles, J., and Field, Z. (2016). Discovering statistics using R. Thousand Oaks, California: SAGE Publications. Fraser J., Goswami U., & Conti-Ramsden G. (2010). Dyslexia and specific language impairment: the role of phonology and auditory processing. Science Studies of Reading, 14(l), 8-29. doi.org/10.1080/10888430903242068 Friberg, J. & McNamara, T. (2010). Evaluating the reliability and validity of (central) auditory processing tests: A preliminary investigation. Journal of Educational Audiology, 16, 4 17. Frick, S. (2016). To wear or not to wear noise-cancelling headphones. Retrieved from https://vitallinks.com/blog/to-wear-or-not-to-wear-noise-cancelling-headphones/. Geffner, D. (2013). Central auditory processing disorders: Definition, description, and behaviors. In D. Geffner & D. Ross-Swain (Eds.), Auditory processing disorders: Assessment, management, and treatment (2nd ed., pp. 59-89). San Diego, CA: Plural Publishing Inc. Ghanizadeh, A. (2011). Sensory processing problems in children with ADHD, a systematic review. Psychiatry Investigation, 8(2), 89-94. http://doi.org/10.4306/pi.2011.8.2.89 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 57 Goberville, E., Beaugrand, G., Hautekete, N., Piquot, Y., & Luczak, C. (2015). Uncertainties in the projection of species distributions related to general circulation models. Ecology and Evolution, 5(5), 1100-1116. doi:10.1002/ECE3.1411 Gonzlez Salinas, R., Garca Gutirrez, D. G., Garcell, J. R., & Hernndez-Montiel, H. L. (2014). Clinical approach in neurodevelopmental delay screening tests. Archivos De Neurociencias, 19(4), 190-193. Gourley, L., Wind, C., Henninger, & Chinitz, S. (2013). Sensory processing difficulties, behavioral problems, and parental stress in clinical population of young children. Journal of Children and Family Studies, 22(7), 912-921. doi:10.1007/s10826-012-9650-9 Green, B. N., & Johnson, C. D. (2015). Interprofessional collaboration in research, education, and clinical practice: Working together for a better future. Journal of Chiropractic Education, 29(1) 1-10. doi. 10.7899/Jce-14-36 Gyldenkaerne, P., Dillon, H., Sharma, M., & Purdy, S. C. (2014). Attend to this: Relationship between auditory processing disorders and attention deficits. American Journal of Audiology, 25(7), 676-687. doi:10.3766/jaaa.25.7.6 Hansen, K. D. & Jirikowic, T. (2013). A comparison of the sensory profile and sensory processing measure home form for children with fetal alcohol spectrum disorders. Physical & Occupational therapy in Pediatrics, 33(3), 440-452. doi:10.3109/01942638.2013.791914 Hind, S. E., Haines-Bazrafshan, R., Benton, C. L., Brassington, W., Towle, B., & Moore, D. R. (2011). Prevalence of clinical referrals having hearing thresholds within normal limits. International Journal of Audiology, 50, 708-716. doi: 10.3109/14992027.2011.582049 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 58 Hosmer, D.W. & Lemeshow, S. (1989) Applied Logistic Regression. John Wiley & Sons, Inc., New York. James, K., Miller, L., Schaaf, R., Nielsen, D., & Schoen, S. (2011). Phenotypes within sensory modulation dysfunction. Comprehensive Psychiatry, 52(6), 715-724. doi:10.1016/J.COMPPSYCH.2010.11.010 Jimenez-Valverde, A., & Lobo, J. M. (2007). Threshold criteria for conversion of probability of species presence to either-or-presence-absence. ACTA Oecologica, 31, 361-369. doi:10.1016/j.actao.2007.02.001 Keith, R. W. (2009). Technical report: SCAN-3 for children. Retrieved from http://images.pearsonclinical.com/images/pdf/technical_reports/SCAN3C_TechReport.p df. Keith, R. W. & Farah, R. (2013). SCAN-3: An evidence-based diagnostic test battery for auditory processing disorders. In D. Geffner & D. Ross-Swain (Eds.), Auditory processing disorders: Assessment, management, and treatment (2nd ed., pp. 141-157). San Diego, CA: Plural Publishing Inc. Kent State University (2019). SPSS: Chi square test of independence. Retrieved from https://libguides.library.kent.edu/SPSS/ChiSquare. Kozou, H., Azouz, H. G., Abdou, R. M., & Shaltout, A. (2018). Evaluation and remediation of central auditory processing disorders in children with autism spectrum disorders. International Journal of Pediatric Otorhinolaryngology, 104, 36-42. doi:10.1016/J.IJPORL.2017.10.039 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 59 Kreisman, B. M. & Crandell, C. C. (2002). Frequency modulated (FM) systems for children with normal hearing. Retrived from www.audiologyonline.com/articles/frequencymodulation-fm-systems-for-1166. Kuczynski, B., & Kolakowsky-Hayner, S. A. (2011). Auditory discrimination. In J. S. Kreutzer, J. DeLuca, & Caplan B. (Eds.), Encyclopedia of Clinical Neuropsychology. New York, NY: Springer Publisher. Lane, S. J. & Reynolds, S. (2019). Sensory over-responsivity as an added dimension in ADHD. Frontiers in Integrative Neuroscience, 13(40). doi: 10.3389/fnint.2019.00040 Lane, S. J., Reynolds, S., & Thacker, L. (2010). Sensory over-responsivity and ADHD: Differentiating using electrodermal responses, cortisol, and anxiety. Frontiers in Integrative Neuroscience, 4(8), 1-11. http://doi.org/10.3389/fnint.2010.00008 Linke, A. C., Jao Keehn, R. J., Pueschel, E. B., Fishman, I., & Mller, R. (2018). Children with ASD show links between aberrant sound processing, social symptoms, and atypical auditory interhemispheric and thalamocortical functional connectivity. Developmental Cognitive Neuroscience, 29, 117-126. doi:10.1016/J.DCN.2017.01.007 Lortie, M., Proulx-Bgin, L., Saint-Amour, D., Cousineau, D., Thoret, H., & Lepage, J.-F. (2017). Brief Report: Biological Sound Processing in Children with Autistic Spectrum Disorder. Journal of Autism and Developmental Disorders, 47(6), 19041909. https://doi-org.proxy01.shawnee.edu/10.1007/s10803-017-3093-5 Lucker, J. (2015). Auditory processing disorder under IDEA. The ASHA Leader, 20(5), 4. doi:10.1044/leader.IN1.20052015.4 Matsuzaki J, Kagitani-Shimono K, Sugata H, Hanaie R, Nagatani F, Yamamoto T, Tachibana M, Tominaga K, Hirata M, Mohri I, & Taniike M (2017) Delayed Mismatch Field Latencies CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 60 in Autism Spectrum Disorder with Abnormal Auditory Sensitivity: A Magnetoencephalographic Study. Frontier of Human Neuroscience. 11, 446. doi: 10.3389/fnhum.2017.00446 McIntosh, D. N., Miller, L. J., Shyu, V., & Dunn, W. (1999). Overview of the Short Sensory Profile (SSP). In W. Dunn (Ed.). Sensory profile manual (pp. 59-73). San Antonio, TX: Psychological Corporation. Miller L., Anzalone, M., Lane S., Chermak, S., & Osten, E. (2007). Concept evolution in sensory integration: A proposed nosology for diagnosis. American Journal of Occupational Therapy, 61, 135140. doi:10.5014/ajot.61.2.135 Miller, L. J., Nielsen, D. M., & Schoen, S. A. (2012). Attention deficit hyperactivity disorder and sensory modulation disorder: A comparison of behavior and physiology. Research in Developmental Disabilities, 33, 804-818. doi. 10.1016/j.ridd.2011.12.005 Miller, L. J., Reisman, J. E., McIntosh, D.N., Simon, J. ( 2001). An ecological model of sensory modulation. In S. Smith Roley, E. I. Blanche, & R. C. Schaaf (Eds.). Understanding the nature of sensory integration with diverse populations. U.S.A.: Therapy Skill Builders. Miller, L. J., Schoen, S., & Mulligan, S. (2016). Sensory Processing 3-Dimensions Measure: Research edition, unpublished test manual. Mieses, A., Tavassoli, T., Li, E., Soorya, L., Lurie, S., Wang, A., & ... Kolevzon, A. (2016). Brief report: Sensory reactivity in children with Phelan-McDermid Syndrome. Journal of Autism & Developmental Disorders, 46(7), 2508-2513. doi:10.1007/s10803-016-2754-0 Mller, A. R., Kern, J. K., & Grannemann, B. (2005). Are the non-classical auditory pathways involved in autism and PDD? Neurological Research, 27(6), 625629. Retrieved from CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 61 http://search.ebscohost.com.proxy01.shawnee.edu/login.aspx?direct=true&db=mnh&AN =16157013&site=eds-live&scope=site Mundkur, N. (2005). Neuroplasticity in children. The Indian Journal of Pediatrics, 72(10), 855857. doi:10.1007/BF02731115 Narkhede, S. (2018). Understanding AUC-ROC curve. Retrieved from https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5 National Early Childhood Technical Assistance Center (2011). The importance of early intervention for infants and toddlers with disabilities and their families. Retrieved from www.nectac.org/~pdfs/pubs/importanceofearlyintervention.pdf. Newsom, J. (2017). Logistic regression. Retrieved from http://web.pdx.edu/~newsomj/pa551/lectur21.htm Ocak, E., Eshraghi, R. S., Danesh, A., R., Mittal, R., & Eshraghi A. A. (2018). Central Auditory Processing Disorders in Individuals with Autism Spectrum Disorders. Balkan Medical Journal, 5, 367. https://doi-org.proxy01.shawnee.edu/10.4274/balkanmedj.2018.0853 Ohio Department of Education. (2014). Ohio operating standards for the education of children with disabilities. Retrieved from https://education.ohio.gov/getattachment/Topics/Special-Education/Federal-and-StateRequirements/Operational-Standards-and-Guidance/2014-Ohio-Operating-Standards-forthe-Education-of-Children-with-Disabilities.pdf.aspx. Ohio Department of Health (2015). 2015 hearing screening guidelines, and requirements. Retrieved from https://odh.ohio.gov/wps/portal/gov/odh/know-our-programs/children-shearing-vision-program/requirements/2015-guidelines. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 62 Ohl, A., Butler, C., Carney, C., Jarmel, E., Palmieri, M., Pottheiser, D., & Smith, T. (2017). TestRetest reliability of the sensory profile caregiver questionnaire. American Journal of Occupational Therapy, 66, 483-487.http://dx.doiorg/105014/ajot.2012.003517 Orekhova, E. V., Tsetlin, M. M., Butorina, A. V., Novikova, S. I., Gratchev, V. V., Sokolov, P. A., Stroganova, T. A. (2012). Auditory Cortex Responses to Clicks and Sensory Modulation Difficulties in Children with Autism Spectrum Disorders (ASD). PLoS ONE, 7(6), 110. https://doi-org.proxy01.shawnee.edu/10.1371/journal.pone.0039906 Otto-Meyer, S., Krizman, J., White-Schwoch, T., & Kraus, N. (2018). Children with autism spectrum disorder have unstable neural responses to sound. Experimental Brain Research, 236(3), 733743. https://doi-org.proxy01.shawnee.edu/10.1007/s00221-0175164-4 Palfrey, T. D., & Duff, D. (2007). Central auditory processing disorders: Review and case study. Axon L'axone, 28(3), 20-23. Parham, L. D., Ecker, C., Miller-Kuhananeck, Henry, D. A. & Glennon, T. (2007). Sensory Processing Measure (SPM) Manual. Los Angeles: Western Psychological Services. Parham, L. D., & Mailloux, Z. (2015). Sensory integration. In J. Case-Smith & J. C. OBrian (Eds.), Occupational therapy for children and adolescents (7th ed., pp. 258-302). St. Louis, Missouri: Elsevier Mosby. Pedroli, G. (2019). ROC curves. Retrieved from http://www.sonoworld.com/TheFetus/page.aspx?id=1698. Pfeiffer, B., Daly, B. P., Nicholls, E. G., & Gullo, D. F. (2014). Assessing sensory processing problems in children with and without attention deficit hyperactivity disorder, Physical & Occupational Therapy in Pediatrics, 35(1), 1-12. doi:10.3109/01942638.2014.904471 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 63 Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Application to practice (3rd ed.). Upper Saddle River, N.J: Pearson/Prentice Hall. Positive Behavioral Interventions & Supports (2019). Tier 2. Retrieved from https://www.pbis.org/pbis/tier-2. R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing: Vienna, Austria. Retrieved from http://www.R-project.org/. Reynolds, S., & Lane, S. J. (2007). Diagnostic validity of sensory over-responsivity: A review of the literature and case reports. Journal of Autism Developmental Disorder, 38, 516-529. doi.10.1007/s10803-007-0418-9 Reynolds, S., Kuhaneck, H. M., & Pfeiffer, B. (2016). Systematic review of the effectiveness of frequency modulation devices in improving academic outcomes in children with auditory processing difficulties. American Journal of Occupational Therapy, 70(1), 7001220030p1-7001220030p11. doi:10.5014/ajot.2016.016832 Rice, C. E., Van Naarden Braun, K., Kogan, M. D., Smith, C., Kavanagh, L., Strickland, B., & Blumberg, S. J. (2014). Screening for developmental delays among young children -National Survey of Children's Health, United States, 2007. Morbidity & Mortality Weekly Report, 63, 27-35. Retrieved from https://www.cdc.gov/mmwr/preview/mmwrhtml/su6302a5.htm. Rogers, S., Hepburn, S., & Wehner, E. (2003). Parent reports of sensory symptoms in toddlers with autism and those with other developmental disorders. Journal of Autism & Developmental Disorders, 33(6), 631-642. doi.10.4306/pi.2011.8.2.89 Ross-Swain, D. (2013). The speech-language pathologists role in the assessment of auditory processing skills. In D. Geffner & D. Ross-Swain (Eds.), Auditory processing disorders: CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 64 Assessment, management, and treatment (2nd ed., pp. 251-282). San Diego, CA.: Plural Publishing Inc. Royeen, C. B. & Fortune, J. C. (1990). TIE: Touch inventory for elementary-school aged children. American Journal of Occupational Therapy, 4, 155-159. Russell, G., Norwich, B. & Gwernan-Jones, R. (2012). When diagnosis is uncertain: variation in conclusions after psychological assessment of a six-year-old child. Early Child Development and Care, 182(12), 1575-1592. doi:10.1080/03004430.2011.641541 Salinas, R., Gutierrez, D. G., Garcell, J. R., & Hernandez-Montiel, H. L. (2013). Clinical approach in neurodevelopmental delay screening tests. Archives of Neurosciences, 18(4), 190-193. doi:10.1186/1471-2431-12-72 Schafer, E. C., Mathews, L., Mehta, S., Hill, M., Munoz, A., Bishop, R., & Moloney, M. (2013). Personal FM systems for children with autism spectrum disorders (ASD) and/or attention-deficit hyperactivity disorder (ADHD): An initial investigation. Journal of Communication Disorders, 46, 30-52. doi.org/10.1016/j.jcomdis.2012.09.002 Schell, B. A., & Cervero, R. M. (1993). Clinical reasoning in occupational therapy: An interactive approach. American Journal of Occupational Therapy, 47(7), 605-610. Schoen, S. A., Miller, L. J., & Flanagan, J. (2018). A retrospective pre-post treatment study of occupational therapy intervention for children with sensory processing challenges, The Open Journal of Occupational Therapy, 6(1), 1-14. https://doi.org/10.15453/21686408.1367 Schoen, S. A., Miller, L. J., & Sullivan, J. C. (2016). The development and psychometric properties of the Sensory Processing Scale Inventory: A report of sensory modulation. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 65 Journal of Intellectual & Developmental Disabilities, 42(1), 12-21. http://dx.doi.org/10.3109/13668250.2016.1195490 Shapiro, Z. (2016). Dont wait to diagnose auditory processing disorder. The ASHA Leader, 21, 34-35. doi:10.1044/leader.SCM.21122016.34 Sharma, M., Purdy, S. C., & Kelly, A. S. (2012). A randomized control trial of interventions in school-aged children with auditory processing disorders. International Journal of Audiology, 51(7), 506-518. doi:10.3109/14992027.2012.670272 Shield, B., & Dockrell, J. (2008). The Effects of classroom and environmental noise on childrens academic performance. 9th International Congress on Noise as a Public Health Problem (ICBEN), Foxwoods, CT. Smith, S. (2019). Ardelis. Retrieved from https://ardalis.com/the-more-you-know-the-more-yourealize-you-dont-know. Statistic Solutions (2019). Chi square test of independence. Retrieved from https://www.statisticssolutions.com/non-parametric-analysis-chi-square/ Swet, , J. (1988) Measuring the accuracy of diagnostic systems. Science, 240, 1285-1293. Tape, T.G. (2015) Interpreting diagnostic tests: The area under an ROC curve. Retrieved from http://gim.unmc.edu/dxtests/roc3.htm. Tazeau, Y. N., & Hamaguchi, P. M. (2013). Disorders and deficits that co-occur or look like APD. In D. Geffner & D. Ross-Swain (Eds.), Auditory processing disorders: Assessment, management, and treatment (2nd ed., pp. 91-116). San Diego, CA: Plural Publishing Inc. Tomchek, S., & Dunn, W. (2007). Sensory processing in children with and without autism: A comparative study using the Short Sensory Profile. American Journal of Occupational Therapy, 61(2), 190-200. doi:10.5014/ajot.61.2.190 CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 66 Trochim, W. M. K., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Mason, OH : Thomson Custom Publications. ISBN: 978-1-592-60291-9. Tyler, R. S., Pienkowski, M., Roncancio, E. R., Jin Jun, H., Brozoski, T., Dauman, N.,Moore, B. C. (2014). A review of hyperacusis and future directions: Part 1. Definitions and manifestations. American Journal of Audiology, 23, 402-419. doi: 10.1044/2014_AJA14-0010 Ubiali, T., Sanfins, M. D., Borges, L. R., & Colella-Santos, M. F. (2016). Contralateral noise stimulation delays P300 latency in school-aged children. PLoS ONE, 11(2), 1 14. https://doi-org.proxy01.shawnee.edu/10.1371/journal.pone.0148360 University of Connecticut (2019). T-tests. Retrieved from https://researchbasics.education.uconn.edu/t-test/# Updike, C. D. (2006). The use of FM systems for children with attention deficit disorder. Journal of Educational Audiology, 13, 7-14. http://www.edaud.org/journal/2006/1-article-06.pdf Van Hulle, C. A., Schmidt, N. L., & Goldsmith, H. H. (2012). Is sensory over-responsivity distinguishable from childhood behavior problems? A phenotypic and genetic analysis. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 53(1), 6472. http://doi.org/10.1111/j.1469-7610.2011.02432.x White-Schwoch, T., Carr, K. W., Thompson, E. C., Anderson, S., Nicol, T., Bradlow Kraus, N. (2015). Auditory processing in noise: A preschool biomarker for literacy. PLoS Biology, 13(7), e1002196. doi:10.137/journal.pbio.1002196 Williamson, G. G., & Anzalone, M. E. (2001). Sensory integration and self-regulation in infants and toddlers: Helping very young children interact with their environment. Washington DC: Zero to Three. CHILDREN AT-RISK OF SENSORY AND AUDITORY DEFICTS 67 Wikipedia (2018). Sensory processing disorder. Retrieved from https://en.wikipedia.org/wiki/Sensory_processing_disorder. Yochman, A., Alon-Beery, O., Sribman, A., & Parush, S. (2013). Differential diagnosis of sensory modulation disorder (SMD) and attention deficit hyperactivity disorder (ADHD): Participation, sensation, and attention. Frontiers in Human Neuroscience, 7(862), 1-10. http://doi.org.proxy01.shawnee.edu/10.3389/fnhum.2013.00862 Zero to Three (2016). DC: 0-5: Diagnostic classification of mental health and developmental discords of infancy and early childhood. Washington, DC: Author.  ...

... CONFLICTS OF INTEST AND RESEARCHER ALLEGIANCE IN CLINICAL TRIALS OF DRY NEEDLING FOR MUSCULSKELETAL PAIN DISORDERS: A SYSTEMATIC APPRAISAL Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: John W. Ware, PT, MS Copyright June 25, 2020 By: John W. Ware, PT, MS All rights reserved Approved by: Stephanie Combs Miller, PT, PhD Committee Co-chair Heather Freeman, PT, DHS, CSCS Committee Co-chair Thomas Hoogeboom, PT, PhD Committee Member David Scalzitti, PT, PhD Committee Member __________________________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program Chair, Interprofessional Health & Aging Studies University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis Running head: CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS Conflicts of Interest and Researcher Allegiance in Clinical Trial of Dry Needling for Musculoskeletal Pain Disorders: A Systematic Appraisal John Ware University of Indianapolis 1 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 2 Abstract Concerns over potential bias due to conflicts of interest (COI) have gained increased attention in the biomedical and psychotherapy literature in recent years. However, little formal analysis of COI in clinical trials of physical therapy (PT) interventions has occurred. As in psychotherapy, PT interventions are often characterized by complex treatment rituals that influence the clinician-patient interaction and therefore the outcome of treatment. One such intervention that has gained significant popularity among physical therapists is dry needling (DN) to treat pain and disability due to musculoskeletal disorders (MSDs). In the psychotherapy literature, a form of non-financial COI has been described as researcher allegiance (RA), which occurs when a clinician-researcher demonstrates a preference for an intervention based on a belief in its superiority over other treatments. The purpose of this study was twofold: 1) to determine the frequency and methods of COI in published DN trials, and 2) to determine the frequency of RA and the nature of reported COI in DN trials and their association. A systematic search of the literature was undertaken to identify trials of DN published between January 1, 2013 and July 16, 2018. Two independent reviewers extracted COI and RA data from published reports. In addition, authors were contacted to obtain information on funding in reports that lacked this information. Finally, study authors were sent a survey inquiring about the presence of RA, and these responses were compared to RA items identified in those study reports combined with a random subset of 14 reports. Sixteen systematic reviews contained 62 unique trials of DN for MSDs. Only 56% per cent of DN trials had a COI statement and just 37% had information on funding in the report. Just 1 report disclosed a potential COI; therefore, no association between COI and RA could be determined. A post hoc analysis showed only 5% of journals in which DN trials were published were members of the International Committee of Journal Editors, which requires comprehensive COI reporting. Authors from 20 (32%) DN trials responded to the RA survey. At least 1 RA item was present in 100% of DN trials according to the surveys but in just half of reports. A nearly 5-fold greater magnitude of RA per trial was found in the survey responses (3.75/5) compared to study reports (0.80/5). The disparities in COI and RA reporting in published trials compared to survey responses suggest that COI and RA might be under-reported in DN trials. Trials of CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 3 psychotherapy interventions have found significant effects of RA on outcomes, and the effect increases with the magnitude of RA. Improved reporting of COI/RA in clinical trials of PT interventions may improve trustworthiness of results and help identify the various factors involved in complex interventions provided by physical therapists. Doing so, could help optimize PT treatments. Keywords: conflict of interest, researcher allegiance, dry needling, systematic appraisal CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 4 Acknowledgements I am particularly grateful to my wife, Kristen Ware, PT, who provided steadfast and patient support during this long journey to complete my doctorate. I would not have been able to get this project off the ground, much less completed, without the assistance of Dr. Thomas Hoogeboom, whose unwaveringly honest critique and advice during the entire process kept this project on task and on point. Im also indebted to the rest of my committee members: for the capable and dogged support of my co-chairs, Dr. Stephanie Miller and Dr. Heather Freeman to see this project through to the end, and finally to Dr. David Scalzitti, whose later arrival on the project was invaluable in adding critical guidance on the design. I also want to express my gratitude to Dr. Greg Manship, former UIndy IRB Director, whose early input on the ethical aspects of this topic were critical in shaping my understanding of some dense philosophical and moral concepts. And lastly, I want to acknowledge Dr. Kenny Venere and Dr. Kyle Ridgeway, two highly dedicated clinical PTs, who acted, respectively, as my 2nd and 3rd reviewers. Kenny, in particular, spent considerable time and effort extracting data from the dry needling reports. CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 5 Table of Contents Title Page 1 Abstract... 2 Acknowledgements. 4 Table of Contents. 5 Chapter 1: Introduction 7 Background. 7 Problem Statement. 10 Study Objectives 11 Operational definitions... 11 Chapter 2: Literature Review.. 13 Section 1 Development of medical ethics in Western medicine 14 Section 2.0 COI in the biomedical research literature... 20 Section 2.1 Historical predominance of financial COI in clinical practice and research.. 24 Section 2.2 Non-financial COI in health care research...... 26 Section 2.3 COI in physical therapy research 32 Section 3 Dry needling in physical therapy research and practice........................ 36 Chapter 3: Methods. 41 Study design... 41 Literature search and study selection. 41 Screening procedures. 42 Data extraction... 43 Chapter 4: Results. . 47 Search results. 47 Characteristics of DN trials.47 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 6 Methodological quality of DN trials. 48 Funding information in DN trials.. 48 Conflicts of interest reporting in DN trials 50 Researcher allegiance rating in DN trials.. 50 Chapter 5: Discussion. 52 Chapter 6: Conclusion 57 References.. 58 Tables. 75 Table 1. Characteristics of meta-analyses of randomized clinical trials investigating effects of needling therapy on pain for musculoskeletal pain disorders.. ............. 75 Table 2. Needling therapy companies offering courses to physical therapists.. 76 Table 3. General study characteristics................................................................................... 77 Table 4. General characteristics of included dry needling trials stratified by funding source.. 78 Table 5. Rating of researcher allegiance in dry needling studies.. 78 Figures. 79 Figure 1. The Healthcare professional behavior Cycle.............................................................. 79 Figure 2. Causal chain of impaired professional judgment................... 79 Appendices.. 80 Appendix 1. PRISMA flow chart............................... 80 Appendix 2. COI classification definitions............................ 81 Appendix 3. Researcher allegiance survey............... 82 Appendix 4. Membership in publication guidelines organizations of journals in which dry needling trials were published.. 89 Appendix 5. In-depth discussion on study quality and funding source. 90 Appendix 6. List of Dry Needling Trials....................... 92 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 7 Conflicts of Interest and Researcher Allegiance in Clinical Trial of Dry Needling for Musculoskeletal Pain Disorders: A Systematic Appraisal In the medical,15 dental,6 and psychotherapy710 research literature, numerous investigators have raised concerns about the influence of conflicts of interest (COI) on the results and conclusions in clinical trials and other empirical studies of interventions in their respective fields. The concern is that various manifestations of COI, both financial and non-financial in nature, may bias the results and conclusions from clinical trials and subsequent systematic reviews,1117 which then influence clinicians to adopt treatments that lack an adequate scientific basis or may even be harmful. While the issue of COI has gained wide attention in other health care professions, a similar systematic effort to address the presence and effect of COI in the research of non-invasive, conservative treatments for musculoskeletal pain disorders (MPDs) provided by physical therapists has not been undertaken. Now that the physical therapy (PT) profession has advanced to the clinical doctorate as its entry-level degree, with a commensurate emphasis on evidence-based practice, theres a need to ensure that interventions provided by physical therapists are based on research that can be trusted. Therefore, more systematic and rigorous scrutiny of the potential bias resulting from COI in research relevant to physical therapy practice is needed. The efficacy and effectiveness of conservative treatments for a wide range of MPDs remains unclear.1824 An example of an intervention for MPDs that has gained popularity among non-medical providers, including physical therapists, in recent years is dry needling (DN). However, two recent meta-analyses of DN to treat MPDs, published in the same journal just a few years apart, reached opposing conclusions about the efficacy of this intervention for reducing pain and disability.25,26 While Kietrys et al25 ascribed grade A evidence to DN and recommended its use in clinical practice to treat neck pain, Gattie et al26 found less favorable CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 8 results for a variety of MPDs treated by physical therapists and did not recommend adoption of DN into physical therapist practice. When research evidence is presented in the form of metaanalyses, which are weighted heavily as a warrant for adoption or rejection of an intervention, clinicians may be less likely to apply evidence to practice if such studies conflict. Investigations of practice patterns in other health care professions show that conflicting results and unclear or ambiguous research reporting are major barriers to implementation of evidence-based practice.27 29 Rapid increases in clinical interest and research of DN have occurred in recent years. The increase in clinical interest has contributed to the proliferation of DN continuing education programs. A recent rapid expansion of DN courses for physical therapists has occurred since the late 1990s (Table 1), with most of these programs appearing within the last 5 years. A Medline/EBSCO Host search using the terms dry needling OR intramuscular stimulation from 1947 through 1999 yielded only 48 citations. However, from 2000 through October 2018, the number of citations using the same terms has increased nearly 10-fold in this single database alone. Finally, DN has received endorsements from professional physical therapy organizations in the United States. The Federation of State Boards of Physical Therapy, the American Physical Therapy Association, and the American Academy of Orthopaedic Manual Physical Therapists have published position statements which support the use of DN by physical therapists.30,31 In order to have confidence in the reported clinical benefits of DN and minimize the risk of harm, including undue financial burden to patients, we believe it is vital to have a robust and thorough understanding of the validity of the evidence base in terms of the classic risks of bias (i.e., randomization, blinding, allocation concealment, loss to follow-up, intention to treat analysis, etc.) as well as the often overlooked impact of COI. Although previous meta-analyses have CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 9 addressed the efficacy and effectiveness of DN, 25,26,3236 an assessment using a consistent methodology to account for the presence and potential influence of COI on DN trials has not been attempted. Furthermore, assessing only the classical elements regarding risk of bias might be insufficient. According to the findings from a recent series of large-scale Cochrane reviews,2,37 industry sponsorship of drug and medical device trials led to more favorable results for the drug or device under investigation, suggesting the influence of financial COI on outcomes. However, industry-sponsored studies did not have significantly higher risk of bias than nonindustry-sponsored studies. The authors concluded that a separate source of bias, not accounted for by the traditional risk of bias domains, must be present in studies that are prone to COI. The conventional understanding of COI is that potential financial gain unduly influences an investigators objectivity, which can then result in bias in the design and reporting of results of clinical trials. In pharmaceutical drug and medical device studies, this occurs when the trial is sponsored by the industry that manufactures and markets the drug being investigated.2,3,38 However, in addition to financial COI, recent methodological analyses of empirical studies show a growing concern over the influence of non-financial COI,8,11,3941 and this is reflected in a recently updated definition of COI cited in an Institute of Medicines committee report for managing COI in clinical practice guidelines42(p78): 1) A divergence between an individuals private interests and his or her professional obligations such that an independent observer might reasonably question whether the individuals professional actions or decisions are motivated by personal gain, such as financial, academic advancement, clinical revenue streams, or community standing, and 2) A financial or intellectual relationship that may impact an individuals ability to approach a scientific question with an open mind.43(p565) CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 10 In the case of DN, the proliferation of continuing education programs teaching these methods warrants concern of direct financial COI inasmuch as these private business entities may employ faculty who are involved in clinical research of DN. Also, as with drug and device company sponsorship of medical research, funding of DN clinical trials by companies involved in manufacture and sale of needling equipment and supplies constitutes potential financial COI. A manifestation of non-financial COI has been conceptualized in clinical research of psychotherapies as researcher allegiance (RA). 810,14,15,4446 Therapeutic allegiance was the term originally used by Luborsky et46(p1003) al in 1975 to describe clinical researchers that are partisans of a form of treatment who do studies of it. The concept developed out of an increasing awareness that clinical trials of a variety of psychotherapeutic interventions for the same condition showed positive results- the so-called Dodo bird effect.4547 Subsequent analysis has shown that indeed RA has an independent and robust influence on the results of psychotherapeutic treatments.15,44,48 However, whether positive results reflect an association between researchers level of training and proficiency with the method or a bias in the design and implementation of the studies has been an ongoing topic of debate among researchers in the field of psychology.4951 As of this writing (June 2020), we are not aware of any formal effort to address the issue of RA as a form of non-financial COI and potential source of bias in the efficacy research investigating treatments for patients with pain due to MPDs. With respect to non-financial COI/RA in DN research, significant growth in studies reporting positive results of DN for a variety of MPDs has taken place over the last decade (Table 2). As positive results of outcome trials accumulate, study authors are more likely to over-estimate effects sizes of subsequent meta-analyses of the treatment.52 The potential for professional investigators and the academic institutions they work in to derive at least indirect CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 11 non-financial benefits from DN research makes the investigators and institution susceptible to COI and could therefore bias the studies results and conclusions. Based on these concerns, the aims of this study are twofold: 1. To determine the frequency and methods of conflict of interest (COI) reporting in published DN trials. 2. To determine the frequency of researcher allegiance (RA) and the nature of reported COI in DN trials and to assess their association. Operational definitions Conflict of Interest: 1) A divergence between an individuals private interests and his or her professional obligations such that an independent observer might reasonably question whether the individuals professional actions or decisions are motivated by personal gain, such as financial, academic advancement, clinical revenue streams, or community standing, and 2) A financial or intellectual relationship that may impact an individuals ability to approach a scientific question with an open mind.43(p565) Researcher Allegiance: in the context of treatment outcomes research, is a belief in the superiority of a treatment [and]usually also entails a belief in the superior validity of the theory of change that is associated with the treatment.49(p55) This allegiance produces an effect separate from any efficacy of the treatment, itself. Dry Needling: use of a solid, thin filiform needle to penetrate the skin and stimulate underlying myofascial trigger points, muscular, and connective tissues for the management of neuromusculoskeletal pain and movement impairments.30(p2) Musculoskeletal Pain Disorder: a condition characterized by impairments in the neuromusculoskeletal system resulting in pain, reduction in range of motion, and disability. CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 12 Common examples include low back pain, shoulder tendinopathy, and knee osteoarthritis. Studies that enroll participants with acute, subacute and chronic MPDs will be included. Conditions treated with DN, such as fibromyalgia, migraine headache, post-operative pain, and medical conditions related to suspected visceral organ disease and psychological disorders were excluded from this investigation. CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 13 Literature Review Between 1995 and 2011, biomedical research funding tripled in the United States, with similar amounts originating from federal government and private industry sources.53 With such a prodigious increase in funding opportunities for scientific research, the influence of conflicts of interest (COI) on the reporting and outcomes of research studies has gained increased scrutiny. A recent issue of the Journal of the American Medical Association (JAMA) was entirely devoted to the issue of COI in medical research, education and practice.54 Concerns over the influence of COI in other health professions have recently become a topic of formal investigation, as well,610,38 but not nearly to the degree or intensity that has occurred in the medical profession. Nonetheless, the PT profession has taken notice of growing COI concerns in health care research, practice and education. Prompted by the above-mentioned JAMA issue, the editor-in-chief of the American Physical Therapy Association-sponsored journal Physical Therapy, Allan Jette, recently penned an editorial addressing the issue of COI in PT research,16 and he followed that up more recently with another commentary specifically addressing the issue of spin in the publication of clinical trials in PT.17 Prior to this, in 2010, Jettes predecessor as editor-in-chief of Physical Therapy, Rebecca Craik, announced the journals adoption of the COI standards put forth by the International Committee of Medical Journal Editors.55 Both editors addressed financial and nonfinancial forms of COI. Craik broke down the sources of COI into four major categories: 1. Authors associations with commercial entities that provided support for the work reported in the submitted manuscript. 2. Authors associations with commercial entities that have an interest in the general area of the submitted manuscript. 3. Any financial associations involving the author or authors spouse and children younger than 18 years of age. 4. Any nonfinancial associations that may be relevant to the submitted manuscript.55 13 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 14 Subsequently, Jette expressed his assessment that concerns of COI have disproportionately focused on secondary financial interests, and then he provided a more detailed breakdown of the nonfinancial conflicts that are also potential sources of bias: 1. 2. 3. 4. Friendships Institutional affiliations Previous work in a given topic area Interest in professional advancement16 In the more recent editorial published in Physical Therapy, Jette highlighted the issue of spin in reports of clinical trials, whereby these incentives produce an inherent temptationfor researchers to report the findings of their research in the most favorable light.17 Jettes comments echo the most recent Agency for Healthcare Quality in Research report Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions, wherein spin is defined as biased presentation of results in the discussion and conclusion of individual studies that should be considered a flag for potential risk of bias.56 Jette cited data from multiple studies showing the prevalence of publication spin in clinical research, and then proposed strategies to minimize this source of bias. However, the prevalence data cited was from the biomedical research. To date (June 2020), no formal effort has been undertaken to examine this issue in clinical research specifically relevant to PT practice. The purpose of this review is to examine the development of the concept of COI within medical ethics, and more particularly in the field of PT and the various ways COI influences the conduct and interpretation of clinical research. In particular, the rapidly expanding and controversial practice of DN in PT will be presented as an object lesson of the influence of COI in producing potential bias in the reporting and conclusions of clinical trials. Section 1 Development of medical ethics in Western medicine 14 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 15 It is not surprising that formal analysis of the influence of COI has far and away predominated in medicine and biomedical research. The origin of medical ethics can be traced back to antiquity, most commonly associated with Hippocrates and the Hippocratic Oath, but also with his contemporary, Aristotle. A physician himself, Aristotle provided the basis for modern medical ethics with his descriptions of the character traits of the virtuous practitioner.57 Aristotles virtuous traits were summarized into two categories by Pellegrino57: Medical Virtues Fidelity Honesty and truth-telling Compassion Effacement of self-interest Courage Justice Intellectual Virtues Science Art Practical wisdom Intuitive wisdom Theoretical wisdom These traits were further advanced by the Pythagoreans in the Hippocratic texts and continued to undergo metamorphosis as they were shaped by the Greek Stoics and then primarily, particularly in the West, Judeo-Christian precepts.58 However, despite these ancient origins, defining the professional role of the modern physician within a moral framework has by no means proceeded along an incrementally positive course towards the attainment of the virtuous traits described by Aristotle. McCullough59 recounted how medicine had by the 18th century regressed to a genuine marketplace of medical practice in which a hodgepodge of often marginally-trained practitioners vied for a chance at treating illness and injury with poorly evidenced interventions. In a previous paper addressing the ethical challenges facing physician-leaders during the advancement of managed health care networks, Chervenak, McCulloughs co-author, warned that a modern-day crisis was occurring in medicine that parallel[s] the situation in eighteenth century Great Britain.60(p880) McCullough seemed to attribute this deterioration in ethical conduct on the ancient Hippocratic texts themselves, arguing that they promote as a primary goal protection of the self-interests of physicians over the interests of patients. 15 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 16 Furthermore, due to the lack of a formal scientific method acquired during the Enlightenment, McCullough argued that the Hippocratic ethic fundamentally lacked moorings in a clearly defined concept of professionalism.61 According to these authors, medicine as a profession was not defined in a modern sense until the mid-18th century when John Gregory and Thomas Percival invented the concept.59,61 Orr et al,62 in contrast, referred to the development of medical professionalism as a kind of moral archeology63(p377) that evolved out of the same foundational precepts that gave rise to the morals and values of Western Civilization. The results of these investigators content analysis of medical schools in the United States and Canada showed that nearly half of them continued to use some form of the Hippocratic Oath up to 1993, thus demonstrating the endurance of Hippocratic ideals into the modern era. Jotterand63(p114) agreed when she stated, Clearly, the [Hippocratic] Oath contains moral and ethical obligations, prohibitions, and exhortations that constitute the professions ideology behind Hippocratic medicine. Moreover, she added that those who have diminished the role of the Hippocratic Oath in shaping modern medical practice discount the full force of its power as a document to direct professional conduct.64(p115) Regardless of when exactly medical professionalism originated, the rise in the use of some form of the Hippocratic Oath during the last century, and in particular since the 1960s,62 suggests a general sense that whatever behavior constitutes a moral and ethical practitioner continues to be a major concern among physicians, and this concern has more recently been taken up by other health care professions as well, including PT. The crisis these authors warn of is a moral one that starts with a threat to the fiduciary relationship between the provider of care and the patient. As Pellegrino stated, The obligations of physician as physician, the first step in medical morality, must depend on what we think of the healing relationship (emphasis added). The unique nature of the healing relationship is defined by the unwell patient who arrives for care in a wounded state of humanity and therefore is built on vulnerability and a promise. 64 More pragmatically, however, Brody65(p264) defined the fiduciary 16 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 17 relationship as one in which one holds something in trust for another. Brody drew on a legal definition of fiduciary: the relationship between two individuals wherein one is empowered to act in the best interests of the other. Typically, this definition applies to situations in which fiduciaries exert control over property owned by an individual, or agent, and therefore have a duty to be loyal and they must account for their actions;66(p347) although, it fails to explain how the fiduciary relationship differs, if at all, in the caregiver-patient and investigator-study participant roles. Many other authors in the area of medical ethics and bioethics have referred to the fiduciary relationship between physician and patient.57,60,61,6769 Indeed, Chervenak and McCullough60(p876) define the concept of medical professionalism as a physician who is the moral fiduciary of the patient. Erde68(p188) defines this relationship in no uncertain moral terms as a doctors primary moral virtueto be trustworthy as the unfailing champion of his patients life and health. Bion67(p586) referred to the essential fiduciary relationship between physician and patient as a key element, which demand[s] that we demonstrate the highest possible standards of objectivity and respect for the truth in scientific research, education and clinical practice. Bions67 definition serves as a reminder that the fiduciary duty is role-based, depending on the social context in which the health care practitioner is operating. Erde70 described the complex interaction of the various social roles assumed by physicians as they join into a care-provision relationship with a patient. He went on further to break down the features of health care practitioners social roles that are susceptible to COI: 1. They are socially designed and elected, in contrast to those that seem natural and unavoidable. 2. They exist to serve the welfare or vital interests of others. 3. They involve discretion and judgment as part of the role holders function. 4. Either the beneficiaries of the role holders work or society in general must be able to trust the role holder simply because he holds the role. The relationship betweenthe professional and his client crucially involves the professionals trustworthiness.70(pp24-25) 17 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 18 Thus, the various roles held by the health care professional (HCP) can be depicted as a spoked wheel with the clinician-patient role as the hub of the wheel (Figure 1). The HCPs duties and responsibilities are assumed within distinct personal and public roles: as a care provider for the patient; as the member of a health care profession to hold each member accountable for breaches of the public trust via an ethical code of conduct and the legal mechanism of professional practice statutes; as the member of the particular health care industry (e.g. the business of providing physical therapy care) in which the clinician makes a living to engage in ethical business practices, including due diligence and honesty in billing and marketing of services; as the case may be, the role of clinical investigator to advance scientific knowledge and improve the quality of care through clinical research.71 Thus, the fiduciary relationship establishes the ethical grounding upon which the professional role of a health care practitioner is based, which in turn directs moral behavior no matter which role the professional currently occupies. For instance, as an investigator, the Investigator-Study Participant role shifts to the hub with the Clinician-Patient role becoming peripheral along with the Profession-Public and Industry-Public roles. However, none of the roles ever becomes irrelevant, as they all encompass what it means to function as a professional practitioner of healthcare. The focus of this review is on the research investigator role, in which researchers serve as fiduciary to both study participants and the public. Which interests are primary in the context of clinical research, however, remains a topic of intense debate. Those promoting the difference position maintain that the roles of clinician-patient and investigator-participant are governed by separate and distinct ethical principles.72 Essentially, advocates of the difference position argue that clinician-investigators do not have the same duty to 18 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 19 optimize therapeutic benefit (therapeutic obligation) for study participants as caregivers have for patients (i.e. the similarity position) because the goals are different. In the former case, the goal is to advance medical knowledge to benefit future patients whereas the caregivers fidelity rests with providing care to the patient that is consistent with the principles of beneficence and nonmaleficence.73 Indeed, Miller and Weijer,74 proponents of the similarity position, acknowledged that the goal of clinical research is to advance the public interest for the benefit of future patients, but that this represents a competing interest to the duty of care to the patient-participant. Moreover, they argued that the provision of optimal care to study participants is misleading because the duty of care does not require provision of the best care available, only that competent care, according to the existing standard of care, is required.74,75 However, as noted by Shamoo,76 often there is no or scant evidence to support even a standard of care upon which competent care can be established. Therefore, the underlying ethical foundation upon which competent care is situated, clinical equipoise (a state of honest, professional disagreement in the clinical community regarding the preferred treatment for a given condition),77 is rendered scientifically invalid because the comparison is made from different levels of evidence, such as randomized control trial (RCT) versus case report, and therefore violates the tenets of evidence-based medicine.78 London79 also agreed with Shamoo76 that as an underlying normative assumption of clinical equipoise, the therapeutic obligation does not apply in clinical trials that enroll healthy subjects. London80 has attempted to reconcile these divisions by proposing an integrative approach that avoids weighing the common good (e.g., advancement of scientific knowledge) against the interests of the individual (e.g., maintenance or restoration of good health) based on some utilitarian ethical framework. Indeed, one of the most renowned proponents of the difference position, Franklin Miller,81 has expressed the need for a coherent moral identity of physician investigators, in which [T]he roles of clinician and scientist are integrated to manage conscientiously the ethical 19 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 20 complexity, ambiguity, and tensions between the potentially competing loyalties of science and care of volunteer patients.81(p1449) London discussed the need to have equal regard for the individual interests of study participants and the societal interests that are served by advancing medical knowledge through scientific research. The individual good and common good refer to these two distinct sets of interests. Each individual has an interest in health that is not only provided within the caregiver-patient context but is also subserved by advancing scientifically sound medical research.80 However, London, in agreement with those who are critical of including medical practice and research under the same ethical umbrella,8286 stipulated that invoking a formulation of clinical equipoise based on a therapeutic obligation to study participants requires a level of individual clinician-researcher intervention that is paternalistic. For example, if an individual decides to forego treatment for side effects during a clinical trial in order to improve medical knowledge for future patients, then it would be a usurpation of that study participants autonomy to provide a treatment that is known to effectively treat the side-effects. Under a therapeutic obligation requirement, a physician would be obligated to provide the current standard of care. Section 2.0 COI in the biomedical research literature The foregoing discussion lays out the ethical concept of health care as a profession wherein practitioners serve as fiduciaries to patients, study participants and the public in order to advance the two sets of basic interests represented by the individual and common good. The onus, therefore, is on HCPs, both individually and corporately, to first identify and then effectively manage situations that potentially elevate self-interest above their duty to patients and research subjects. Within the context of research, the concern is that the failure to perform ones professional duty due to COI risks bias in the reporting and conclusions of clinical trials. In fact, increased concerns over COI in biomedical research is supported by an expanding body of empirical evidence showing significant bias resulting from COI,1,2,5,37,87,88 prompting efforts 20 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 21 to more clearly define and describe COI so that it can be better identified and managed. The ultimate goal is to minimize bias so that reporting of results and conclusions from clinical trials can be trusted to inform and advance practice. The National Academy of Medicine89 (formerly the Institute of Medicine) recently cited a more comprehensive definition, which was proposed by the American Thoracic Society in 2009: a relationship that may place primary interests (e.g., public well-being or research integrity) at risk of being improperly influenced by the secondary, personal interests of the relationship (e.g., financial, professional, or intellectual gains).43 This definition, however, has engendered disagreement over what actually constitutes COI. In particular, the inclusion of non-financial secondary interests that, by definition, compete with the primary interests of research integrity, patient care and education of students is an ongoing and at times contentious debate. This debate was featured in a recent JAMA series devoted entirely to the topic of COI.54 Specifically, Bero90 argued that this category of secondary interests do not constitute a true COI, referring to those as non-financial interests and discounting their importance relative to the potentially more insidious influence of financial COI. Its worth noting that the vast majority of the commentaries in the JAMA issue focused primarily on COI related to financial incentives and business arrangements. Beros piece was particularly notable in that she specifically excluded other competing interests (e.g. personal beliefs, intellectual/academic background, desire for professional prestige) that arent based on a personal relationship or financial motive as capable of producing a COI. Moreover, in a commentary with Grundy, Bero argued that the negative effects of COI from personal relationships rarely extend beyond the immediate situation as opposed to the widespread and harmful source of bias resulting from financial COI. 91(p4) Others have rejected the attempt to separate financial and non-financial COI into distinct categories. Wiersma and colleagues92 argued that parsing COI into separate categories risks diminishing the potential negative influence they can have on research and clinical practice because 21 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 22 they over-simplify strategies and create artificial parameters to manage them. They referenced the research in psychology and sociology showing that non-financial incentives can influence human attitudes and behaviors at least as much as financial ones. Their definition of COI acknowledges the multi-factorial complexity of clinical practice, including overlapping and intertwined motives, values, incentives and ethical demands of treating patients and advancing clinical practice. Moreover, under the increased pressures of applying evidence-based practice, clinical decisions need to be made according to the complex judgments used to integrate empirical evidence from clinical trials with patient values and preferences.93 These authors reference and subscribe to Erdes definition of COI, which acknowledges the complex interaction between motives and social arrangements that can unduly influence the responsibilities of HCPs to observe, judge, and act according to the moral requirements of their role.70(p33) An example of this more nuanced definition of COI was provided by Larkin and Loewenstein94 in the JAMA issue devoted to COI54 wherein the authors described the inherent issues of COI within existing health care business models, which range from a tendency towards overutilization in fee-for-service models to under-utilization in plans employing capitated payment schemes. However, they noted that the relationship between payment incentives (or disincentives) is hard to disentangle from the complexities of clinical decision-making: A complicating factor is the difficulty of assessing whether any individual procedure was influenced by physician incentives; ie, of measuring bias at the procedure level. Medical care involves significant uncertainty and heterogeneity in treatment efficacy, and patients also vary in their needs and preferences. These complexities make it difficult or impossible to identify specific cases in which payments influenced decisions, which both increases the potential for conflicts of interest to occur and makes it impossible to address the problems with remedies such as the threat of malpractice suits for unwarranted procedures.94(p1746) For instance, is an early decision to order diagnostic imaging for a patient with acute low back pain (LBP) based on a direct financial incentive, an indirect one related to concerns of malpractice, or a 22 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 23 particular aspect of the patients clinical presentation that prompted the clinician to deviate from an established clinical practice guideline? Or, might perhaps even a combination of those factors influence the clinicians decision to order early imaging? While there is a growing body of evidence that early diagnostic imaging for acute LBP results in no better outcomes at higher costs9597 and even worse outcomes in the occupational low back population,98 the reasons for this behavior are not clear. For example, an article published in The Wall Street Journal describing a cost-analysis study at Virginia Mason Medical Center found that physicians who had no apparent or structural financial incentive (i.e., they were salaried employees) to order imaging studies for patients with acute LBP had simply gotten in the habit of ordering magnetic resonance imaging for these patients.99 As Larkin and Loewenstein94 suggested, structural elements within the business models that dominate health care delivery may be driving the habit of choosing low-value care. Moreover, since decision-making, even moral decisions involving the care of patients, are subject to automatic/intuitive reasoning processes, which are driven by self-interest, theres always a risk that even a seemingly minor COI can result in the escalation of unethical behavior. Deliberative/ controlled processes are more likely to be associated with ethical judgments.100 This might explain how even nominal gifts from pharmaceutical salespeople, like pens and notepads, can have a significant influence on the prescribing behavior of physicians.101 With the inherent complexity and intermingling of various forms of COI in mind, its important to appreciate that the risks of failing to effectively manage COI can result in impaired judgments by researchers and clinicians (Figure 2). Therefore, it is imperative that HCPs take measures pre-emptively to identify, prioritize and limit COI in any forms it may take. The concern that bias will create an impaired judgment in the validity of research claims or clinical decisions has been borne out by empirical research in several health care disciplines.1,2,38,44,48,52,102,103,510,14,15 23 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 24 Formal, published evidence of harm to subjects or patients as a result of these biases is less clear, with several notable exceptions.103106 Section 2.1 Historical predominance of financial COI in clinical practice and research Historically, financial COI has garnered most of the attention in both the scientific and, perhaps even more influentially, the major news media. The empirical evidence showing the negative influence of financial COI on the impartiality of industry-sponsored clinical trials became so compelling that in 1995, the U.S. government instituted regulations on financial COIs through the Department of Health and Human Services and the Public Health Service. The regulations define both who qualifies as an investigator subject to the rules and what constitutes a significant financial interest,53 which is defined as: anything of monetary value, whether or not the value is readily ascertainable, that Is related to the Investigators professional responsibilities on behalf of the Institution including, but not limited to, activities such as research, research consultation, teaching, professional practice, institutional committee memberships, and service panels Belongs to the investigator or the investigators spouse or dependent children53 The recently published issue of JAMA54 devoted entirely to COI contained only two articles that addressed non-financial forms of COI. The remaining over two dozen articles primarily focused on the problem of financial COI in clinical practice, research and education. Its possible that highlypublicized news media attention to incidents resulting in large-scale harm to the public health or in which research subjects were potentially harmed or killed has directed more attention on the negative influences of financial COI. Analyses of the soft drink industrys influence on the research agenda has produced widespread media attention showing that companies like Coca Cola suppressed data suggesting sugar plays a role in chronic diseases, such as coronary heart disease and diabetes.108110 However, the most common target of the media when it comes to reporting on financial COI is the pharmaceutical industry. In recent years, several major news organizations have reported on various forms of payments to physicians to promote a drug or medical device companys products.111117 24 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 25 These news stories recount details of harm caused to study participants and patients while researchers, clinicians and institutions conducting the treatments and investigations received significant payments from the sponsoring large corporation. In addition to news stories, books by highly-regarded and influential physician leaders with provocative titles like On The Take: How Americas Complicity with Big Business Can Endanger Your Health118 and The Truth About the Drug Companies: How They Deceive Us and What to Do About It119 promote a narrative that COI is synonymous with dubious financial incentives that are bound to result in compromised clinical judgments. While theres no question that direct financial incentives often underlie threats to HCPs primary duty to patients, a sales tactic that has come under scrutiny recently is white coat marketing.116,117,120 In the cases reported, a nurse served dual roles as a marketer of a particular companys pharmaceutical product while at the same time engaging with patients as a medical educator. Such a role would be easy for the nurse to justify on the grounds that a valuable health care-related service is being provided to patients; however, the fact that the fiduciary duty as caregiver competed with the nurses role as salesperson for the drug company creates a COI. A particular example of white coat marketing, however, is instructive of the complexity of overlapping roles and the incentives associated with them. Bloomberg Businessweek reported on a drug that was developed to treat extremely rare kidney disorders, thus earning its inclusion as an orphan drug.120 These drugs are typically extremely costly for the person with the rare condition primarily because they are so expensive to develop and the population of people with the condition is so small. Therefore, the drug companies are under tremendous pressure to recoup their investment by finding individuals with the condition and then ensuring their compliance with the treatment protocol, which is why they hire field nurses to directly interact with patients. The story includes the case of a patient who reported feeling highly pressured by a nurse to remain on the drug even though she was not receiving significant benefits. The patient reported that the nurse warned her that she may suffer 25 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 26 lethal blood clots, a common complication of the rare kidney disease, if she discontinued the drug. In this particular case, unlike some other cases of white coat marketing, the patient was well-aware that the nurse worked for the pharmaceutical company. The article was unclear about what other financial incentive the nurse may have received by keeping the patient on the drug. However, information was provided that during sales meetings nurses were pressured by the sales team to keep patients on the drug by warning them of potential lethal consequences and, if necessary, helping them find another physician who would put them back on the drug. The obvious conclusion from the interaction with the sales team is that the pressure placed on the nurses was driven by financial selfinterests, but certainly the nurses could have also experienced social pressures from their work colleagues to be part of the team. Thus, casting this scenario as simply a financial COI, in which no other competing interests influenced the nurses behavior, is an example of Erdes Artificially Narrow Account: [COIs] occur when and only when a physician strays or is tempted to stray from his role-mandated duties for the sake of his own economic benefit.70(p13) The pharmaceutical companys solution for managing COI was to have their nurses report to medical affairs instead of sales. While eliminating the avoidable financial COI from direct contact with the sales force, it is possible that nurses who are reporting to medical colleagues, particularly physicians, will experience a different sort of social, role-related pressure that could compete with their duty to the patient. Section 2.2 Non-financial COI in health care research The broader definition of COI offered by Erde70 promotes a more subtle and complex, but important potential source of bias that could impede efforts to advance clinical practice, research and education. These more subtle and varied competing interests may pose more threats to unbiased clinical judgements in the context of treating complex health conditions due to the fact that they can be so difficult to identify and then manage.11,67,92,121 Moreover, as several have noted, financial and non-financial COI are often so intermingled that they are scarcely distinguishable.67,92,100,101 For 26 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 27 instance, in addition to the direct financial benefit in the event the physician owns the imaging equipment, the example of ordering early imaging for acute low back pain (LBP) could be motivated by an earnest and clinically important desire to reduce the anxiety of a distressed patient or caregiver, despite the fact that early imaging has failed to show improved outcomes.9597 Alternatively, that decision could also be motivated by a combination of conscious and/or unconscious competing interests, such as the clinician being pressed for time and therefore not being able or willing to spend the extra time needed to reduce the patients or caregivers anxiety by providing a more thorough explanation of the favorable natural history of acute LBP. Other competing interests based on the clinicians values, such as her sense of duty to waiting patients, or practical concerns (e.g. fatigue) could also influence the ultimate decision to order imaging. This practical example shows that making a determination of which incentive- an aversion to rudeness or the increased revenue from a superfluous imaging study- drove the clinicians behavior in any given clinical decision-making scenario consists of the interaction of often several over-lapping factors. How could these possibly be disentangled without imputing the clinicians motive? Wadman122 warned that when a charge of COI narrowly directs attention to financial arrangements and incentives, which then leads to an imputation of the physician-investigators motive, this can plunge public debates into moralistic blame games and lead to the withholding of information that could inform the debate.123(p3) Numerous studies in the biomedical and health care-related research have investigated the presence and influence of non-financialCOI.3,7,46,48,52,123,124,810,1315,44,45 Unsurprisingly, the growth of empirical data showing undue influence of non-financial competing interests has promoted this form of COI as a general topic of concern on the editorial pages of many HCP journals.3,50,67,91,107,121,125138 However, both large and subtle differences remain in how various types of non-financial COI are defined. 27 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 28 Guyatt et al125 defined intellectual COI as academic activities that create the potential for an attachment to a specific point of view that could unduly affect an individuals judgment about a specific recommendation.126(p739) However, it is important to note that this definition was made in the context of clinical practice guideline (CPG) development, which consists of a panel of experts in the particular field of academic study or clinical practice that is tasked with the development and recommendation of the guideline. Due to the normative implications of CPGs, they are having a broad and increased influence on clinical practice4,139 and can also impact reimbursement patterns.140 Therefore, to the extent CPG development and implementation includes a variety of individuals with differing interpretations of the evidence, those individuals may be subject to closer scrutiny for COI by a variety of stakeholders than other forms of evidence, such as clinical trials and systematic reviews. Since CPGs are developed from a systematic review of best evidence, most notably the efficacy and effectiveness research, it would make sense to obviate undue influence of COI by managing it effectively at the individual study level before the data is evaluated by the CPG panel and preventing the problem of garbage in, garbage out. There is evidence that CPGs are subject to COI and that CPG recommendations have been negatively influenced by forms of COI that could not be defined as strictly financial in nature. A 2012 survey141 found that less than half of the 37 organizations that frequently issue CPGs even had a COI policy in place, and of those that did about half of them failed to meet any of the National Academy of Medicines COI standards.42 A study of 456 World Health Organization CPGs found that a majority of strong recommendations were based on low- or very low-quality evidence,142 and a subsequent qualitative analysis of panelists reasons for making strong recommendations showed that political considerations and unwarranted certainty of benefits from an intervention influenced some panelists decisions.143 28 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 29 Empirical evidence over a period of several decades has shown that a particular form of intellectual COI, referred to as researcher allegiance, has resulted in measurable bias in the outcomes of psychotherapy trials.810,14,15,4446 Therapeutic allegiance is a term originally defined by Luborsky et al46 in 1975 to describe clinical researchers that are partisans of a form of treatment who do the studies of it.46(p1003) In that and subsequent research,9,45 Luborsky and colleagues went on to show that the so-called Dodo bird verdict, wherein clinical trials comparing interventions by researchers who possessed an allegiance to the method of therapy, showed an independent positive effect on the outcome due to the allegiance. The Dodo bird verdict is a reference to the character in Lewis Carrolls classic Alice in Wonderland, who at the end of a race that measured only a single dependent variable (clothes drying after running around a lake for various distances and times) announces, Everyone has won and all must have prizes. The implication being that apart from the therapists preference for the particular method, the interventions resulted in equivalent outcomes. The concept was first proposed by Rosenzweig47 about 40 years prior to the initial formal analysis by Luborsky et al46 Rosenzweig theorized that different psychotherapeutic methods would produce similar results because non-specific effects of the interaction between patient and therapist are more important than any particular attribute of the method itself.47 Within the context of the current financially strained U.S. health care delivery system, where care providers are reimbursed according to a system of coding procedural units of treatment, all stakeholders in the health care system, HCPs in particular, should be concerned about the awarding of prizes to practitioners. Over the last decade, several commentaries in prominent medical and medical ethics journals have addressed the problem of over-utilization in the U.S. health care system related to procedural reimbursement,144146 and recent empirical analyses have confirmed that offensive medicine is widespread.147,148 An analogous situation may be occuring in clinical research, in which grant prizes are indiscriminately awarded to those investigators who report 29 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 30 positive results of trials, thereby impeding appropriate allotment of funding resources to the most promising research questions that minimize risks to patient-participants. A plethora of evidence over the last several decades confirms the presence of publication bias, whereby positive trial results are much more frequently reported than negative trials.149152 It is as yet unclear what role the allegiance effect may play on publication bias, but it stands to reason that a clinical researcher who has an allegiance to a certain treatment model- one which she uses to treat patients and experiences clinical success- would be more likely to favorably review a clinical trial of that method thereby increasing its likelihood of publication. There is empirical evidence that indirectly supports this conclusion. Panagiotou and Ioannidis52 showed that researchers who have published positive trials of an intervention are significantly more likely to believe that a subsequent meta-analysis of that intervention will show stronger effect sizes on the primary outcome than methodologists, who more accurately estimated the effects sizes of the intervention. The concept of equipoise was first articulated by Fried153 to provide an ethical underpinning for conducting RCTs but also to mitigate the effects of researcher bias, including preference or allegiance to a particular intervention. Freedman77 refined Frieds concept with the term clinical equipoise, which he defined as a state of genuine uncertainty within the expert medical communityabout the preferred treatment.78(p1) According to this definition, no arm of the trial is accepted by the clinical community as superior to any other nor is one treatment preferred by all expert clinicians in a given area of practice, thereby protecting study participants interests but also presumably minimizing the effects of a treatment preference or allegiance that could produce bias in the outcomes of clinical trials. If true uncertainty exists regarding optimal care for a particular condition, then a RCT is warranted because the honest null hypothesis pre-condition has been met.86 However, as discussed earlier when contrasting the similarity and difference position, contention over the validity of the concept of equipoise centers on a profound disagreement between 30 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 31 the role of the care provider and clinical researcher. Those who argue that equipoise is a valid concept contend that it is founded on the clinician-investigators duty to provide competent care during a clinical trial while also ensuring that important clinical questions are addressed.74,82,154 Whereas those who reject the validity of equipoise counter that the ethics of clinical research and practice are distinct, 72,73,82,86 and the notion that the roles are similar is based on therapeutic misconception; that is, the mistaken belief that the primary reason for a patient enrolling in a clinical trial is to gain therapeutic benefit. Opponents of equipoise argue that the primary goal of clinical research is the advancement of knowledge to help future patients while preventing exploitation of study participants.73,83 With respect to concerns of COI, the question arises: Does the concept of clinical equipoise, grounded in the duty of care or therapeutic obligation, effectively protect the interests of study participants as the primary interest of the researcher? As elucidated by London,79 and consistent with criticisms of equipoise,76,86 the equipoise condition lacks applicability to research contexts that include healthy volunteers. Perhaps more importantly, however, with respect to guarding against COI, is the problem of paternalism when an individual clinician is not uncertain about which treatment a patient should receive for a particular condition. According to a formulation of equipoise that is grounded in the duty of care, the clinician would be required to refuse enrollment of a patient into a clinical trial in which the patient is randomized to an intervention that the clinician is certain is inferior. The problem of paternalism arises because the uncertainty is located in the mind of the individual clinician as opposed to the clinical community (thus a strict interpretation of Frieds theoretical equipoise). As van der Graaf and van Delden155 explain, however, this represents confusion over the dilemma that equipoise is intended to resolve. The individual clinician who lacks equipoise when it exists in the clinical community (as may be confirmed by an Institutional Review Board [IRB]) is in a state of disagreement with clinical experts. There are many reasons why such 31 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 32 conflict could exist between the beliefs of an individual clinician and the expert community, one of which includes the influence of COI producing conscious or unconscious bias on the part of the individual clinician, such as might exist if the clinician researcher has an allegiance to the intervention. Section 2.3 COI in physical therapy research Scarce mentions of issues related to COI have appeared in the PT literature. The first article formally addressing COI in PT research appeared in the journal Physical Therapy in 2010 by then Editor-in-Chief, Dr. Rebecca Craik.55 She took that opportunity in an editorial to indicate that the journal was adopting the International Committee of Medical Journal Editors (ICMJE) (www.icmje.org) format for reporting and disclosure of COI. Overall, however, formal study of the topic has been extremely limited. Moreover, the same level of concern about COI that has occurred in biomedicine has yet to appear in the PT profession on such a wide scale. A recent systematic review by Grundy et al38 investigated the influence of COI on clinical trials in multiple non-physician health professions, but did not include PT. Moreover, despite several decades of clinical use in PT practice, the effectiveness of conservative interventions for a variety of MPDs remains in question;156160 therefore, theres an urgent need to determine if, like in psychotherapy, the Dodo bird verdict has rendered all treatments similar in efficacy and effectiveness. If so, then those interventions that result in the least harm to patients and the public health, both in terms of preventing injury and undue financial burdens, may need to be adopted into CPG recommendations. Furthermore, if some treatments are more beneficial for MPDs, are those differences real or are they due to bias resulting from COI- either financial or non-financial? A recent EBSCO-based search of the MEDLINE and CINAHL databases using the keywords conflict of interest AND physical therapy yielded 42 citations compared with over 2800 citations 32 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 33 when combining COI with medicine. Of the 42 citations, about 25% are related to the issue of physician ownership of PT practices. Only one of the citations is an empirical study that addressed the influence of COI on outcomes for patients receiving conservative treatment by physical therapists,161 and in that trial, the COI was related to the comparative drug treatment, not the interventions provided by physical therapists. However, as mentioned earlier, the current Editor-inChief of Physical Therapy, Dr. Alan Jette, prompted by the previously cited JAMA issue,54 recently reiterated the journals commitment to effective disclosure and management of COI at both the study and editorial/peer review levels.16 Jette16 addressed both financial and non-financial COI in his commentary. Specifically, he expressed concern over the multiple potential sources of COI and identified non-financial sources in particular. He included friendships, institutional affiliations, previous work in a given topic area, or interest in professional advancement56(p775) as potential sources of non-financial COI that could influence the judgment of investigators and those involved in publishing research. To date, no similar editorial position has been published regarding the issue of COI at the preeminent musculoskeletal journal in PT, the Journal of Orthopaedic and Sports Physical Therapy. The limited amount of empirical study that has been performed in PT research has addressed COI from the standpoint of clinician preferences to a treatment as defined by the concept of personal equipoise.135 In these trials, the authors invoke the concept of equipoise to establish the ethical basis for enrolling participants and designing clinical trials to protect patient-participants from harm but also to reduce the risk of biased outcomes to a particular intervention or treatment method. Two clinical trials of manual therapy controlled for the presences of clinician-researcher equipoise on outcomes for patients with LBP.162,163 In both of these trials, the authors used the term personal equipoise, which was essentially equivalent to the definition of the converse of researcher allegiance (RA) advanced by Munder et al:10(p670) a researchers preference for a particular 33 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 34 treatment. However, Cook et al162 made a clear distinction between clinical equipoise, as per Freedmans original definition,77 and personal equipoise. The distinction wasnt as clear in the trial by Bishop et al163 as they seemed to conflate the definitions of clinical and personal equipoise: Another consideration is the state of the clinical equipoise of a provider. Clinical equipoise is defined as genuine uncertainty regarding the efficacy of a particular treatment arm.164(p966) The obvious problem with this latter definition adopted by Bishop et al163 is that a practitioners lack of preference for an intervention is not necessarily the same as the clinical communitys collective uncertainty that one intervention is superior to another, i.e. that an honest null hypothesis exists. As mentioned above, this may simply represent an individual clinicians disagreement with the clinical community, which could be due to a lack of knowledge or because of the influence of a COI resulting in an unwarranted preference for a particular intervention, i.e. RA. For instance, the practitioner may train in an intervention that has not been found to be any better than an alternative, but may prefer the alternative only because of their training. That same clinician may have taught that particular method on the continuing education circuit and therefore have a vested interest in advancing that method or approach to care. Therefore, an attempt to control for personal equipoise may more accurately be considered an effort to manage individual researchers COI. In the Cook et al trial162 comparing thrust to non-thrust mobilization of the spine in patients with mechanical LBP, no difference was found when the authors presumably controlled for clinical and personal equipoise. However, the determination of clinical equipoise seemed to be inconsistent with the actual design of their study. They cited two CPGs as evidence that thrust manipulation has been lauded to improve outcomes and quicker recovery163(p191) for patients with acute LBP. However, the first CPG by Airaksinen et al164 only included recommendations for chronic LBP. It should be noted that the average duration of symptoms in the Cook et al trial was >30 weeks, which is considered well-beyond the typical 7 to 12-week duration of symptoms to meet the standard of 34 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 35 having chronic LBP.165 The second CPG by Laerum et al166 did not distinguish between thrust versus non-thrust spinal manipulation for LBP (acute, subacute, or chronic). Based on these misinterpretations of the evidence upon which this study was conceived, it is questionable that the authors have even established a basis for the existence of equipoise according to any definition or formulation described in the literature. As mentioned earlier, the attempt to control for personal equipoise is more accurately identified as an effort to minimize bias related to RA, which Cook et al indirectly acknowledged when, in reference to the pre-study evaluation of personal equipoise, they stated, This potential bias was controlled within the study.162(p196) The presumption here, however, is that clinicians who preferred one form of joint mobilization for LBP were biased and those who didnt were not. This is a misinterpretation of the concept of equipoise because being agnostic about which intervention is generally more effective for a condition does not necessarily constitute a lack of bias, particularly when that agnosticism is determined a priori with respect to a condition like chronic LBP, which is characterized by a high degree of complexity and clinical variability. It is certainly possible that after evaluating the patient according to the protocol, some clinicians who were found to be in equipoise prior to the study chose to use a certain joint mobilization technique based on a preconceived preference for that particular patient presentation. Indeed, the potential to promote such a bias was built into the design of the study since the thrust techniques used included one originally described by Maitland,167 which presumably targets the spinal segment pain generator, whereas the other technique is generally considered non-specific to a particular level of the spine.168 In fact, this latter technique was originally developed to treat sacroiliac joint dysfunction.169 The fact that a nonspecific thrust technique was utilized in a study protocol that standardized the examination so clinicians would localize the most comparable responseto a specific level of the lumbar spine 35 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 36 suggests that clinicians did in fact resort to a preference bias since one of the techniques used did not comport with the examination protocol. Moreover, determining the presence of bias is not the ethical problem equipoise is intended to address. The purpose of invoking equipoise is to confront the ethical problem of randomizing patient-participants to different arms of a clinical trial due to the uncertainty over which intervention is superior, which then leads to a dilemma of what the clinician-researcher owes the patientparticipant, i.e. the debate over best available care versus competent care. It is the role of IRBs to make a determination of the epistemological soundness of the state of clinical equipoise. In the case of the Cooke et al trial, the IRBs responsibility was to answer the question of whether the authors have adequately justified that valid uncertainty exists among the clinical community over the superiority of thrust and non-thrust mobilization for subjects with mechanical LBP. Ensuring the individual clinician-researchers personal equipoise, on the other hand, is not the relevant requirement for guarding against bias. If an individual clinician-researcher possesses a belief in a treatment that is in conflict with the state of uncertainty in the clinical community, then that individual already presumes to know what best care is despite what the clinical community has concluded. In the Cook et al trial, therefore, it would be unethical for that clinician-researcher to participate, or be allowed to participate, in a trial in which patients were randomized to receive what that clinician-researcher believes to be an inferior treatment. Furthermore, IRBs that evaluate such designs that pre-determine levels of personal equipoise to control for bias need to consider the ethical implications of utilizing clinician-researchers who have already established a preference. What role may COI play in the formation of that preference? Section 3 Dry needling in physical therapy research and practice As Jette16 expressed in his recent editorial, the influence of COI- both financial and nonfinancial- requires more attention in PT research and practice. He noted in particular that concerns 36 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 37 have come to be disproportionately focused on financial COI,56(p775) and identified several other potential sources of COI within the presumably non-financial category. DN is an area of practice that has gained intense interest in PT in recent years. Many systematic reviews of various forms of needling therapy (NT) efficacy trials published over the last decade have shown favorable results on pain in the immediate to medium-term for patients with a variety of MPDs (Table 1). Systematic reviews that included trials of Western forms of acupuncture were included in Table 1 because socalled Western medical acupuncture170 trials are often included in systematic reviews of DN. However, the difference between Eastern and Western forms of NT continues to be elusive and driven by unresolved theoretical distinctions, practice scope debates, and medico-legal issues.171174 Moreover, according to recent analyses, efforts to standardize and improve reporting of NT trials have achieved marginal success in meeting those goals.175,176 Jia et176 al found that less than half of English language trials of NT for knee osteoarthritis reported the theoretical rationale and only about 1% of Chinese language trials did so, thus highlighting the ongoing difficulty trialists encounter in making the distinction between Eastern and Western NT. Although this is not a comprehensive list of NT systematic reviews, Table 1 reflects a wide range of the NT literature from journals accessed by a variety of clinicians who treat MPDs, including PTs. The table also displays the immediate to short-term results of NT on pain, which is consistent with recent guidelines from the American Physical Therapy Association30 stating that NT should be used as part of a broader physical therapy approach28(p5) progressing to active interventions once pain is better managed. Despite the generally favorable- at least in the short- to medium term- results on pain reported in these systematic reviews, several commentaries have recently appeared in the PT literature shedding questions on the validity of these results and the underlying theory,177 misreporting of reliability data in trials of DN for trigger points,178 and the need to question the efficacy of DN;179 all of which could be a reflection of bias in the design, publication and/or reporting of DN trials, as has 37 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 38 been reported in the biomedical literature. Specifically regarding the misreporting of data, in two published clinical trials of DN,180,181 the reliability coefficients for myofascial trigger point diagnosis were erroneously reported indicating much higher levels of agreement than were found in the original study cited by the authors. Each of these studies was approved by the IRBs of different universities, but included two of the same authors and the corresponding author was the same individual in both studies. Rapid increases in clinical interest and research of DN have occurred in recent years. The increase in clinical interest has contributed to the proliferation of DN continuing education programs. Using the Google search terms dry needling continuing education a list was generated of North American DN courses open to physical therapists since the late 1990s, with most of these companies opening within the last 5 years (Table 1). Each course is offered over 3 or 4 days, and the costs reflect more or less typical rates for onsite continuing education courses offered to physical therapists. Although no current standards exists for determining a physical therapists competency in DN based on number of credit hours,182 the amount offered may in part reflect state PT board requirements for licensees. For example, one continuing education company (Mississippi Dry Needling) explicitly references the 50-hour minimum face-to-face requirement to legally utilize NT,183 which is a requirement of both the Mississippi and neighboring Louisiana physical therapist practice acts. The predominant theoretical underpinning for use of DN in PT practice is for treatment of myofascial pain syndrome, or more specifically to inactivate myofascial trigger points.30,184188 However, as described above, other Western science-based mechanisms of action have also been proposed over the course of several decades.189199 Rickards,172 from an osteopathic perspective, attempted to reconcile the most popular forms of NT with the various explanatory models available. However, notably, his analysis172 did not include a reference to the more recently introduced mechanotransduction theory proposed by Langevin,194 which is cited in the APTA Educational 38 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 39 Resource Papers30,200 and in a popularly-cited narrative review by a PT clinical education group.190 Rickards172(p3) made a distinction between Eastern acupuncture systems and modern Western acupuncture systems, and only addressed the latter systems in his analysis since they are based on modern theories of evidence that can be tested through scientific methods of inquiry. That is, they are based on evidence of prior plausibility using the principle of parsimony, and any hypothesis that is generated is compared to the null hypothesis using operationally-defined terms and reliable and valid methods of measurement. Rickards172 identified three treatment models: myofascial trigger point dry needling, neurosegmental acupuncture, and medical acupuncture. These models are largely consistent with other descriptions in the Western literature.179,190,191,195,197,198,201204 Each of the DN continuing education providers listed in Table 2 subscribes to at least one of these explanatory models, in some form. For example, Myopain Seminars strictly adheres to the myofascial trigger point model,186 whereas the Spinal Manipulation Institute invokes all three models plus the more recently proposed theory of mechanotransduction as valid rationales for NT.190 Without question, the term Dry Needling predominates in PT; however, as Rickards and others have pointed out,172174,201 the use of this term has as much to do with medicolegal and practice scope-related issues as any underlying theoretical distinctions. The only published systematic review with meta-analysis of DN performed strictly by physical therapists to date included trials utilizing the myofascial trigger point model.26 Although the results were somewhat mixed, the sensitivity analysis showed that DN was more effective than a validated sham procedure, at least in the short to intermediate (12 weeks) term. However, lower quality evidence showed no difference between DN and other treatments used by physical therapists; including manual therapy, exercise and transcutaneous electrical nerve stimulation. 39 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 40 In the context of potential negative COI influence resulting in biased reporting of results and conclusions of DN trials, these findings should be cause for concern. Recalling Miller and Brodys86 treatise on equipoise and research ethics, they cited the results of psychology trials showing high rates of non-specific, or placebo, effects from treatments for chronic conditions like depression. Nonspecific effects are commonly attributed to the results of manual therapy163,205210 and NT.172,191,197,198,211,212 Moreover, patients with persistent musculoskeletal conditions often utilize PT. Generally speaking, the benefits of PT for persistent pain and disability due to MPDs show small effect sizes.23,156,158,160,213 Therefore, as health care resources become scarcer and pressures increase on clinicians to show value based on the results of the care they provide, it will be imperative that the PT profession design well-controlled trials and systematic reviews with low risk of bias in order to determine the true efficacy and effectiveness of the treatments physical therapists provide. Addressing the scope and influence of COI, as has occurred recently in the biomedical and psychotherapy research, will be a critical part of that process. 40 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 41 Methods This study employed a cross-sectional design. The DN trials (and their respective publishers) in this study were identified by use of a systematic literature search. Therefore, Institutional Review Board approval was not required. Literature search and study selection A systematic search of the scientific literature using PubMed was undertaken to identify systematic reviews and meta-analyses investigating the efficacy of DN for MPDs. Reviews investigating the effects of DN on pain or disability as the primary outcome was included. Only individual clinical trials of DN drawn from systematic reviews/meta-analyses that utilize a published method of assessing methodological quality/risk of bias, such as the Cochrane risk of bias tool and the PEDro quality scale, were included. Trials that compared DN to another treatment, shamcontrolled trials, waiting list controls, and cross-over designs were eligible. Two cohort studies214,215 that appeared in one review216 were included in the analysis. The search string consisted of the following terms combined with the recently updated PubMed filter to identify systematic reviews217: Systematic [sb] AND dry needling [ti] The original scoping search strategy of NT reviews produced 64 systematic reviews that contained 162 unique trials. Screening of the titles and abstracts of those reports showed that numerous clinical trials investigated forms of NT that a) incorporated Eastern principles, b) the rationale was mixed or unclear between Eastern and Western forms of NT, or c) the review included individual trials that investigated both Western and Eastern forms of NT. Since the main focus of this study was to investigate the presence of COI and RA in DN trials most relevant to current PT practice, this targeted and expedited search for systematic reviews of DN was developed. Also, in order to ensure that the DN trials included for analysis were assessed with currently utilized and evidence-based study quality/risk of bias tools, the search was limited to systematic reviews 41 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 42 published since January 1, 2013. Only reviews published in English were included. (See Appendix 1 for the PRISMA flow chart.) Screening procedures Two reviewers, the primary author (JWW) and a second reviewer (KV), performed screening of titles and abstracts using online citation screening software (Rayyan QCRI: www.rayyan.qcri.org218) according to the inclusion and exclusion criteria below. If the two reviewers did not agree, then a third reviewer (KR) was consulted to reach a consensus. Inclusion criteria: Intervention- DN as operationally defined in the Introduction Conditions- MPDs, which is defined as a condition characterized by impairments in the neuromusculoskeletal system resulting in pain, reduction in range of motion, and disability. Common examples include low back pain, shoulder tendinopathy, and knee osteoarthritis. Review of efficacy trials using pain or disability as primary outcome Use of published methodological quality tool to assess risk of bias (per citation in References section of review) Exclusion criteria: Reviews of conditions that do not meet the criteria of a MPD as above, including reviews of experimentally-induced pain in normal subjects and post-operative pain Non-reviews, e.g. review protocols, clinical trials, observational studies, surveys, clinical practice guidelines Reviews that investigated aspects of DN other than efficacy for treatment of MPDs, e.g. mechanism of action, cost-effectiveness, adverse events, utilization Withdrawn studies 42 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 43 Studies of non-human participants The study selection results were depicted in a PRISMA-style flow diagram (Appendix 1).219 Upon completion of the screening of reviews, the primary author extracted all clinical trials from the eligible reviews, added the citations to reference management software, removed duplicate citations, and acquired full text copies of each individual DN trial for data extraction. Data extraction Data extraction from the study reports was completed in three steps: 1) study characteristics, 2) presence of absence of defined COI criteria and 3) presence or absence of operationally defined RA criteria. The primary author (JWW) and second reviewer (KV) performed data extraction. First, the primary author extracted the following study characteristics data from each DN trial report: Number of trial authors; Type of control: sham, another type of treatment (comparative or active control); no treatment (e.g. waiting list or inactive control), Number of participants in each group; Risk of bias tool(s) used to assess methodological quality with numerical score, and Funding source. A random sample of 10% of study reports (n=6) was checked by the second reviewer and returned to the primary author for verification. In addition, if no funding source information was present in the report, the primary author attempted to contact a corresponding author (CA) to inquire about funding for the study. Two methods were used to contact authors: a) the email contact provided in the study report or, if unsuccessful, b) an attempt was made to contact either the CA or primary author at the ResearchGate website (www.researchgate.net). In cases of reviews that did not report the details of study quality/risk of bias criteria, two methods were used to acquire this data. First, the PEDro 43 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 44 website (www.pedro.org.au) was searched to determine if individual scoring data was available for these DN trials (n=7), 5 of which were available at the PEDro website. The remaining 2 trials, which were cohort studies, were scored by the primary author (JWW) using the PEDro scale.220 Finally, if a trial registry was identified in the report, the primary author accessed that information online to determine if a funding source or sponsor of the trial was indicated. Second, both the primary author and the second reviewer independently extracted COI data. After data extraction, the primary author compared results, and a third reviewer was available if needed to resolve any disagreements. The following COI information was extracted from each study: Method of COI disclosure, e.g., narrative statement within report, International Committee of Medical Journal Editors COI disclosure form, online document/supplement Number of authors that report any type of COI Number of authors that report each specific type/subtype of COI (see Operationalizing COI below) Number of authors providing a rationalization statement, e.g., This relationship did not influence the investigators conduct in this trial. (for convenience, the term loogly, coined by Hakoum et al,39 will be used to to identify these statements in the data collection tables and in subsequent references to these statements.) Number of authors with discrepancy between disclosures in the published report and disclosures in other forms, including those provided upon request COI disclosures described as available on request Whether there is a reference to COI disclosure statements for individuals other than authors, such as editors, peer-reviewers, medical writers, others 44 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 45 To operationalize COI, the framework established by Hakoum et al39 was used to initially screen included studies for the various forms of COI that have been shown to unduly influence the reporting of outcomes and conclusions in the empirical research. The table below summarizes this classification framework: Individual Institutional Financial Professional Intellectual Advocatory Individual Financial Institutional Financial Individual Professional Individual Intellectual -- -- -- Institutional Advocatory Based on a recent extensive literature review in the area of COI by the primary author,221 this framework is the only published work to date that comprehensively describes the various forms of COI. The definitions are partly derived from the International Committee of Medical Journal Editors Conflict of Interest Disclosure Form, which is required by each of the six highest impact medical journals for author manuscript submissions. In determining the type of COI present, the definitions developed by Hakoum et al39 to categorize COI in each DN trial (Appendix 2) was used. A pilot test of the COI classifications was conducted by randomly selecting 10% of the included DN trials (n=6), and both reviewers (JWW and KV) determined which type of COI was present based on the data extracted from the manuscript. No disagreements were found between the primary author and second reviewer on which type of COI was present. Finally, to determine the presence of RA, the following set of criteria adapted from Wampold et al48 and Munder et al10 was employed: 1) developed or provided the intervention, 2) trained the clinicians used in the study, 45 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 46 3) supervised the clinicians used in the study, or 4) advocated for the therapy, which is defined as participating as a principle investigator (first, second or last author222) of a previously published positive trial listed in the references section of the article.223 5) Any authors involvement in clinical instruction of the NT method used in the study either within a formal academic setting or on a continuing education course for clinicians was also included as a criterion. The primary author independently reviewed the first 25 trial reports in alphabetical order to determine the presence of any of the RA criteria. A 10% (3 reports) random sample of these 25 studies was created using a random number generator and full-text manuscripts were sent to the second reviewer for analysis. If any discrepancies between the primary author and second reviewer on any RA criterion from the study reports were identified, then the two reviewers met to discuss these differences. A third reviewer was available to consult until agreement was reached on the RA criteria. Additionally, an electronic survey (Survey Monkey, San Mateo, CA) was sent to the CA of each dry needling study (Appendix 3). The survey consisted of questions that address the presence of each of the RA criteria listed above. The survey response period was open from February 21, 2019 until April 23, 2019. 46 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 47 Results Search results The search strategy yielded a total of 17 articles. After title and abstract screening, all 17 articles were selected for full-text screening. The full-text screening resulted in a total of 16 systematic reviews (See PRISMA flow chart in Appendix 1). These 16 systematic reviews comprised 62 separate trials of DN for MPDs that served as the basis for this study (Appendix 6). Nearly half of the DN studies (47%) had been included in one of the 16 systematic reviews, representing the largest proportion compared to studies that appeared in multiple reviews. An average of approximately 15% appeared in 2, 3 or 4 reviews and 10% were included in 5 or more reviews. Characteristics of DN trials The 62 DN trials comprised a total of 3109 patients. The three most studied MPDs were: general myofascial pain syndrome, neck pain, and upper trapezius myofascial pain syndrome. Fiftynine studies (95%) identified trigger points as the target of treatment; although, diagnostic features varied. In one study,224 the authors referred to hypersensitive areas in muscles rather than trigger points as the target of treatment. Sample size ranged from 12225 to 167.226 Comparators for the DN trials were sham/placebo (34%), active control (34%), no control (26%) and inactive control (6%). Of the 16 trials that did not have a control group, 14 were comparative studies that had between two and four experimental groups and, of these, 3 trials compared DN to DN combined with another intervention (A versus A+B). Ten of the 62 DN trials included a reference to an online registry listing. However, the registry record published in one of these studies227 was not retrievable. Table 3 presents the general characteristics of the included DN trials. 47 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 48 Methodological quality of DN trials A summary of quality scores of DN trials in Table 3 shows that nearly a quarter of studies (23%) met less than 50% of the study quality tool criteria; more than half (53%) scored between 50% and 75%; and 21% exceeded 75%. A total of seven DN trials were either not assessed by the review authors (n=2214,215) or the systematic review report did not include a listing or table of the study quality/risk of bias criteria (n=5228232). A breakdown of study quality/risk of bias scores by criteria judged to be unclear or lacking shows that the most commonly missing criterion was blinding of subjects or clinicians (94%) followed by allocation concealment (52%), blinding of assessors (34%), completeness of data (29%) and, lastly, random allocation (23%). Funding information in DN trials Forty-four DN trials (71%) included a statement either in the report or from a CA regarding whether funding was provided for the study. Thirteen DN trials (21%) reported that the study did not receive funding. Thirty-one DN trials (50%) were funded, either internally (n=15), externally (n=15) or both (n=1). The study with both internal and external funding was included in the external funding category in Table 4. Eighteen trials (29%) did not have a statement about funding, and neither a corresponding or primary author could be contacted to retrieve this information. Of the 18 DN trials categorized as not reported, 17 (94%)215,227,231,234246 provided an email address for either a CA or primary author in the report; although, one email address was no longer valid.243 In Table 4 the characteristics of the included DN reports stratified by source of funding (internal, external, non-funded, or not reported) are presented. Nearly half (48%) of funded trials had sample sizes 30 compared to 31% of trials with no funding. Sample sizes <30 occurred in 46% of non-funded studies compared to 19% of funded studies. 48 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 49 The most commonly used control group across DN trials, sham/placebo, was used in 39% of funded studies and 23% of those not receiving funding. Among funded trials, 44% of those with external funding used a sham/placebo compared to 20% of internally-funded trials. Active controls were utilized in 52% of funded trials compared to 38% that were not funded. Active controls were used about twice as frequently in internally-funded DN trials (35%) than those receiving external funding (16%). Thirteen per cent of funded studies compared to 23% of those that were not funded lacked a control group. Eighty-seven per cent of funded DN trials had a study quality score 50% compared to 77% of non-funded studies. No externally funded studies scored <50%, whereas 4 studies receiving internal funding scored <50%, representing 13% of funded trials and 6% overall. A breakdown of study quality/risk of bias criteria by funding source indicates that failure to blind subjects/clinicians occurred most often across funding-source types (ranging between 87% and 100%). Also, lack of subject or clinician blinding occurred with similar frequency in funded (94%) and non-funded (100%) studies. The next most common quality criterion, lack of allocation concealment, occurred in 35% of funded trials compared to 38% of non-funded studies. Among funded studies, nearly twice as many internally funded DN trials (23%) lacked a clear description of allocation concealment compared to those that were externally funded (12%). The third overall most reported quality criterion, non-blinded assessors, occurred in 26% of funded studies and 38% of nonfunded studies. Of those that received funding, internally funded studies had an approximately eightfold greater frequency of non-blinded assessors (23%) than externally funded studies (3%). Incomplete data reporting was the fourth most overall reported study quality criterion, including about one-third of funded studies and 8% of those that were non-funded. Incomplete data reporting occurred with about twice the frequency in internally funded studies (23% of funded studies) compared to those that were externally funded (10%). The least reported overall study quality 49 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 50 criterion, lack of random allocation of study participants, occurred in 19% of funded DN trials compared to 15% of trials that were not funded. Conflicts of interest reporting in DN trials Thirty-four DN trials utilized a narrative statement to disclose COI in the published report resulting in a total of 55% of the DN trials including some form of COI disclosure. The remaining 28 (45%) trials did not include any form of COI disclosure statement in either the published report or online protocol. One study247 made reference to financial disclosure statements within the narrative statement that were obtained from each author; however, no information was included about how to acquire these disclosures. One out of the 34 studies with a COI disclosure made reference to a potential COI or source of funding and listed the funding source in the narrative statement.248 The author was contacted for clarification, and he stated that there was no conflict. Researcher allegiance rating in DN trials After contacting CAs or PAs via the email address provided in the report or through the ResearchGate portal, 20 survey responses (32% response rate) were collected. The first 25 DN reports, in alphabetical order, were assessed for RA by the primary author (JWW), which included 11 studies for which survey responses had been received. Therefore, the sample of RA data consists of 34 DN trials, of which 20 includes data from both the author survey and the report. Table 5 presents the data on rating of RA in the DN studies. The presence of at least one RA criterion was identified in 50% of study reports, and the average number of RA criteria extracted from the reports was 0.80 per study. In contrast, among the 20 studies represented by the survey respondents, 100% reported the presence of at least 1 RA criterion, and the average number of RA criteria was 3.75 per study. 50 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 51 For the first RA criterion developed/provided DN it was found that this item was present more often than not according to the study report (41% to 6%). According to the answers by the survey respondents, 90% of studies had a major author (1st, 2nd or last) provide DN to participants in their study. For the second and third RA criteria trained clinicians and supervised clinicians, the study reports showed that each of these RA factors was present in 3% of DN trials. Based on the survey data, 65% and 80% of these criteria, respectively, were present. For the fourth RA criterion advocated for DN 23% of study reports met this RA criterion. The survey respondents indicated that in 65% of studies a major author was also an author of a previous positive DN clinical trial that was cited in the references of the current study. For the fifth and final RA criterion taught DN course, no information in the study reports suggesting that any of the major authors taught DN in an academic or continuing education course was found. The survey data showed that 75% of survey respondents indicated that an author of the study taught DN as an academic or continuing education course, or both. Three of the CAs participating in the survey identified an author as teaching in an academic course only. The remaining 17 reported that a major author taught in either an academic or continuing education course, or both. 51 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 52 Discussion The aim of this study was two-fold: first, to determine the frequency and methods of COI reporting in published DN trials. In general, low rates of reporting COI in DN studies were found. Forty-five percent of the 62 trials lacked a COI disclosure statement either in the report or at the online registry. Seventy-one percent of the DN trial authors provided information regarding the funding source either in the report itself or through contact with study authors. Of the trials that reported a funding source, 48% were funded internally and 52% externally. The second objective was to determine the frequency of RA, the nature of reported COI in DN trials and to assess their association. Large disparities were found between disclosure of COI (none) and the presence of RA criteria (high). No association could be established due to a potential under-reporting of COI. Of the DN trials with a COI disclosure statement in the published report, none reported a COI. The CA of one study that disclosed a potential COI was contacted via email and stated there was no conflict. The use of the descriptor potential to describe COI has been criticized as misleading and inaccurate because it reflects the view that a COI only exists when bias or harm actually occurs.107 Others have taken exception to the use of the term COI because it is equated with an accusation of research misconduct, when the evidence suggests this rarely actually occurs,249 or that it imputes the motives of researchers causing them to defend what they perceive as unavoidable relationships with funding agencies, including industry sponsors.122 Also, the ongoing debate over what constitutes COI- whether it should be primarily concerned with financial interests related to business and industry relationships 91,94,126,250 or if it should be expanded to include intangible non- financial interests, such as scholarly and ideological positions and desire for prestige and recognition among peers11,12,16,17,40,67,92,128130,134,285288- likely adds to the confusion and defensiveness. That a majority of DN trial reports did not include a COI disclosure and nearly one-third (n=20) did not respond to a minimum of two requests for identification of a funding source, suggests that 52 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 53 impediments to reporting potential sources of bias may be a problem in clinical research of PT interventions. Alan Jette, the editor-in-chief of Physical Therapy, recently expressed his concern that COI is disproportionately focused on financial COI, and this narrow interpretation may at least partly explain the low rate of COI reporting in this study.16 In addition to determining and categorizing COI, RA towards the DN interventions was assessed by two methods: extraction from study reports and solicitation from study authors via a survey questionnaire (Appendix 3). Based on the results from the survey, all studies (100%) had at least some form of RA, and at least 65% of studies scored positive on 3 or more of 5 RA items. In contrast, among the published reports, only half had at least 1 criterion and 2 of the RA items were present in only 1 study. This disparity between DN reports and author surveys suggests RA might be an overlooked issue in the DN literature, which is understandable, as the magnitude of RA was less clear if only the published reports of the studies were assessed. These findings compare with a recent investigation of RA disclosure in psychotherapy trials13 in which about 2/3 of studies were deemed allegiant, and about 12% as non-allegiant. In a related study, this research group8 found a significant effect of RA on study results, and the effect increased with the magnitude of the RA rating. Since RA was first identified as a potential source of bias in psychotherapy trials in 1975,46 study designs have reflected more sophisticated methods to assess its influence.8,9,255,10,14,15,4446,48,123 Given the dearth of RA items present in DN trial reports, it would be difficult to perform similar analyses of the effects of RA on outcomes of DN without the additional step of contacting study authors, which may be subject to errors of author recall and variable interpretation between different authors of the same study. Ostensibly, the wide disparity between RA criteria extracted from the study reports versus survey responses seems to validate a reluctance of authors to disclose COI. However, the findings from the survey seem to belie conventional beliefs that clinical researchers are reticent to confront 53 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 54 potential sources of bias when the issue is framed as RA versus COI, at least in this particular area of PT research. The admission by all 20 respondents to the survey that an average of over three RA criteria was present in their DN trials suggests an eager willingness to disclose relationships to the intervention that could affect clinician behavior during a DN trial and thereby influence results. Furthermore, the nearly 5-fold larger magnitude of RA criteria from the surveys compared to study reports suggests that authors may be unaware of RAs influence on trial designs and outcomes and therefore do not directly address it as a potential source of bias. Noting the lack of established efficacy of psychotherapeutic interventions, Ioannidis51 suggested that allegiance bias should be exploited in clinical trials to provide more information about how to properly provide an intervention and to show if it works. He contended that if the developers and promoters of a new, promising intervention perform the early trials, this could provide important details of how the intervention should ideally be performed. Moreover, if no clinically meaningful effect is found by the experts, then it is unlikely that a benefit will be found in studies by non-allegiant investigators. Therefore, by accounting for what Cook et al162 referred to as adjunctive processes associated with complex interventions, study designs that control for RA may help elucidate how mediators of outcomes; such as therapeutic alliance, natural history and regression to mean, differentially influence treatment effects. Clarification of the contents of the black box of complex interventions may improve the ability to measure other important outcomes; such as costs, adverse events, and recurrence rates, thus increasing the scope of the potential benefits of PT interventions from a broader healthcare delivery perspective. The results of this appraisal of COI and RA in clinical trials of DN for common MPDs inform the clinician-researcher dyad in two important ways: 1) they suggest that framing clinician-researchers treatment preferences in terms of COI may be too narrow a lens to view how these factors influence outcomes. Whereas acknowledging 54 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 55 them in terms of RA, in which clinicians with opposing preferences square-off in wellcontrolled trials with clear operational definitions of RA factors, may provide insight into the complex interactions between clinicians and patients with MPDs and therefore produce more meaningful and trustworthy results, and; 2) they identify possible impediments in the feedback loop between researchers and clinicians that can prevent the growth of clinical practice and the development of more effective treatments for patients with MPDs. The most important recommendation for now from the authors perspective is that journals require authors to clearly and uniformly report COI (including elements comprising RA). Explicit criteria for reporting COI have been provided by the International Committee of Medical Journal Editors (ICMJE). Alas, few of the journals that published DN trials are members. A post hoc analysis to determine the membership status of the journals that published DN trials in organizations that provide guidelines for COI disclosure (Appendix 3) showed that just eight (20%) journals in which DN trials were published are members of ICMJE, and only three of those were members when the DN trials were published. Therefore, presumably only four sets of DN trial authors were required to complete the ICMJE form, which clearly defines the various types of both financial and nonfinancial COI. Finally, the reporting of PT interventions needs to be optimized. Assessments of RA through the screening of the reports showed that this issue was substantially under-reported by DN authors. Half of the studies did not report who developed or provided the intervention. Even more pronounced, under both the trained clinicians and supervised clinicians criteria, the rate of reporting was less than 25%. In contrast, 75% and 80% of survey respondents indicated that there was training and supervision of treating clinicians, respectively, by a major author. It is possible that these elements were left out of the published reports due to space constraints imposed by the journal. The inclusion 55 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 56 of online supplements that describe details of both the treating clinicians and the treatment itself, perhaps based on STRICTA256 or an extension of the TIDieR checklist,257 would provide more opportunities for higher quality investigations into this area of clinical research. Our study has a number of strengths and weaknesses. A strength of our study is that the search strategy yielded a representative sample of DN trials that have been used by physical therapists to inform clinical practice in treatment of MPDs. The 62 trials spanned nearly three decades of DN research, with two-thirds being published since 2010. Half of the DN trials appeared in multiple reviews with about one-third appearing in more than two reviews. Ten (62%) of the systematic reviews, comprising nearly three-fourths of the DN trials, included a physical therapist as a major investigator (first, second or last author). The three most common MPD diagnoses- general myofascial pain, neck pain, and upper trapezius myofascial pain- are commonly treated in PT clinics, and therefore this body of evidence accurately reflects the types of patients that physical therapists treat with DN in current clinical practice. A limitation was the low response rate to the survey. Despite the survey period being open for 3 months with three additional weekly email reminders sent to CAs after the original notice, the response rate from the online RA survey was only 32%. However, 90% of survey respondents were authors of DN trials that were published since 2010, which represents 43% of DN trials analyzed in systematic reviews published since January 1, 2013. Therefore, the survey response data is weighted towards the more recent body of DN clinical research and likely provides a more accurate reflection of the presence of RA in DN trials. Another limitation was that, due to time limitations, only the first 25 DN trial reports in alphabetical order, which included 11 trials from which survey responses were collected, were assessed for RA. Assessment of the entire sample of 62 DN trials may alter the findings, although the alphabetical order likely resulted in a random subset. Finally, a priori modification to one of the RA criteria was made (developed DN was combined with provided DN), and another criterion was added 56 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 57 (taught DN) based on the assumption that providing and teaching an intervention is likely to represent a degree of enthusiasm or endorsement of the treatment. More formal efforts to operationally define RA are needed. Conclusion This systematic appraisal of COI and funding source reporting in clinical trials of DN found that these elements had relatively low rates of reporting. The results of this appraisal suggest that authors of DN trials are quite willing to provide additional details of RA elements in their study designs. However, there seems to be a no one asked about that mindset among authors. Modification or expansion of existing guidelines for ensuring study quality of complex interventions, such as STRICTA256 and TIDieR257, with RA criteria included in online supplements, should be considered to provide a full measure of study methods and then the data used to drive additional research to determine the effects on study quality and results. Furthermore, the inclusion of this data may drive more precise investigations into the mediators of effect sizes of the complex interventions used by physical therapists to treat MPDs. Finally, as the PT profession moves forward with creating a relevant and trustworthy body of clinical research evidence, accurate and complete reporting of COI and RA in clinical trials will improve clinicians understanding of the various factors that can influence study results. More detailed reporting of factors that can influence outcomes will enhance the ability of physical therapists to identify higher quality studies and apply that research to practice. Doing so will promote deepening and strengthening physical therapists fiduciary roles in both the clinic and research labs. 57 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 58 References 1. Bailey CS, Fehlings MG, Rampersaud YR, Hall H, Wai EK, Fisher CG. Industry and evidence-based medicine: Believable or conflicted? A systematic review of the surgical literature. Can J Surg. 2011;54(5):321-326. doi:10.1503/cjs.008610 2. Lundh A, Lexchin J, Mintzes B, et al. Industry sponsorship and research outcome. Lundh A, ed. Cochrane Database Syst Rev. 2017;2(2):MR000033. doi:10.1002/14651858.MR000033.pub3.www.cochranelibrary.com 3. Fries JF, Krishnan E. Equipoise, design bias, and randomized controlled trials: the elusive ethics of new drug development. Arthritis Res Ther. 2004;6(3):R250-5. doi:10.1186/ar1170 4. IOM (Institute of Medicine). Conflict of Interest in Medical Research, Education, and Practice. Washington, D.C.: National Academies Press; 2009. doi:10.17226/12598 5. Printz JO, Lee JJ, Knesek M, Urquhart AG. Conflict of interest in the assessment of hyaluronic acid injections for osteoarthritis of the knee: An updated systematic review. J Arthroplasty. 2013;28(8 SUPPL):30-33. doi:10.1016/j.arth.2013.05.034 6. Beyari MM, Hak A, Li CS, Lamfon H a. Conflict of interest reporting in dentistry randomized controlled trials: a systematic review. J Evid Based Dent Pract. 2014;14(4):158-164. doi:10.1016/j.jebdp.2014.06.002 7. Cristea IA, Gentili C, Pietrini P, Cuijpers P. Is investigator background related to outcome in head to head trials of psychotherapy and pharmacotherapy for adult depression? A systematic review and meta-analysis. PLoS One. 2017;12(2):e0171654. doi:10.1371/journal.pone.0171654 8. Dragioti E, Dimoliatis I, Fountoulakis KN, Evangelou E. A systematic appraisal of allegiance effect in randomized controlled trials of psychotherapy. Ann Gen Psychiatry. 2015;14:25. doi:10.1186/s12991015-0063-1 9. Luborsky L, Diguer L, Seligman DA, et al. The researchers own therapy allegiances: A wild card in comparisons of treatment efficacy. Clin Psychol Sci Pract. 1999;6(1):95106. doi:10.1093/clipsy.6.1.95 10. Munder T, Gerger H, Trelle S, Barth J. Testing the allegiance bias hypothesis: A meta-analysis. Psychother Res. 2011;21(6):670-684. doi:10.1080/10503307.2011.602752 11. Akl EA, El-Hachem P, Abou-Haidar H, Neumann I, Schnemann HJ, Guyatt GH. Considering intellectual, in addition to financial, conflicts of interest proved important in a clinical practice guideline: a descriptive study. J Clin Epidemiol. 2014;67:1222-1228. doi:10.1016/j.jclinepi.2014.05.006 12. Clark AM, Choby A, Ainsworth K, Thompson DR. Addressing conflict of interest in nonpharmacological research. Int J Clin Pract. 2015;69(3):270-272. doi:10.1111/ijcp.12569 13. Dragioti E, Dimoliatis I, Evangelou E. Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal. BMJ Open. 2015;5(6):e007206e007206. doi:10.1136/bmjopen-2014-007206 14. Lieb K, Osten-Sacken J von der, Stoffers-Winterling J, Reiss N, Barth J. Conflicts of interest and spin in reviews of psychological therapies: a systematic review. BMJ Open. 2016;6(4):e010606. doi:10.1136/bmjopen-2015-010606 15. Munder T, Brtsch O, Leonhart R, Gerger H, Barth J. Researcher allegiance in psychotherapy outcome 58 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 59 research: An overview of reviews. Clin Psychol Rev. 2013;33(4):501-511. doi:10.1016/j.cpr.2013.02.002 16. Jette AM. Protecting against bias in PTJ. Phys Ther. 2017;97(8):775-776. doi:10.1093/ptj/pzx059 17. Jette AM. Protecting Against Publication Spin in Clinical Trials. Phys Ther. 2019;99(9):1119-1121. 18. Xia P, Wang X. Effectiveness of ultrasound therapy for myofascial pain syndrome : a systematic review and. J Pain Res. 2017;10:545-555. 19. Page M, Green S, Mrocki M, et al. Electrotherapy modalities for rotator cuff disease ( Review ). Cochrane Database Syst Rev. 2016;(6). doi:10.1002/14651858.CD012225.www.cochranelibrary.com 20. Wegner I, Is W, Mw VT, et al. Traction for low-back pain with or without sciatica. Cochrane Database Syst Rev. 2013;(8). doi:10.1002/14651858.CD003010.pub5.www.cochranelibrary.com 21. Fuentes JP, Olivo SA, Magee J., Gross DP. Effectiveness of Interferential Current Therapy in the Management of Musculoskeletal Pain: A Systematic Review and Meta-Analysis. Phys Ther. 2010;90(9). 22. Jamtvedt G, Dahm KT, Christie A, et al. Physical Therapy Interventions for Patients With Osteoarthritis of the Knee: An Overview of Systematic Reviews. Phys Ther. 2008;88(1):123-136. doi:10.2522/ptj.20070043 23. Keller A, Hayden J, Bombardier C, Van Tulder M. Effect sizes of non-surgical treatments of nonspecific low-back pain. Eur Spine J. 2007;16(11):1776-1788. doi:10.1007/s00586-007-0379-x 24. Feine J, Lund J. An assessment of the efficacy of physical therapy and physical modalities for the control of chronic musculoskeletal pain - University of the Incarnate Word [Alma]. J Int Assoc Study Pain. 1997;71(1):5-23. 25. Kietrys DM, Palombaro KM, Mannheimer JS. Dry needling for management of pain in the upper quarter and craniofacial region. Curr Pain Headache Reports. 2014;18(8):437. doi:10.1007/s11916014-0437-0 26. Gattie E, Cleland JA, Snodgrass S. The Effectiveness of Trigger Point Dry Needling for Musculoskeletal Conditions by Physical Therapists: A Systematic Review and Meta-analysis. J Orthop Sport Phys Ther. 2017;47(3):133-149. doi:10.2519/jospt.2017.7096 27. te Pas E, van Dijk N, Bartelink MEL, Wieringa-De Waard M. Factors influencing the EBM behaviour of GP trainers: A mixed method study. Med Teach. 2012;35(3):e990-7. doi:10.3109/0142159X.2012.733044 28. Knops AM, Vermeulen H, Legemate DA, Ubbink DT. Attitudes, awareness, and barriers regarding evidence-based surgery among surgeons and surgical nurses. World J Surg. 2009;33(7):1348-1355. doi:10.1007/s00268-009-0020-8 29. De Smedt A, Buyl R, Nyssen M. Evidence-based practice in primary health care. Stud Health Technol Inform. 2006;124:651-656. 30. American Physical Therapy Association. Description of Dry Needling In Clinical Practice: An Educational Resource Paper.; 2013. http://www.apta.org/StateIssues/DryNeedling/ClinicalPracticeResourcePaper/. 31. American Academy of Orthopaedic Manual Physical Therapists. AAOMPT Position Statements. Baton Rouge; 2009. 59 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 60 http://aaompt.org/Main/About_Us/Position_Statements/Main/About_Us/Position_Statements.aspx?hke y=03f5a333-f28d-4715-b355-cb25fa9bac2c. 32. Rodrguez-Mansilla J, Gonzlez-Snchez B, De Toro Garca , et al. Effectiveness of dry needling on reducing pain intensity in patients with myofascial pain syndrome: a Meta-analysis. J Tradit Chinese Med. 2016;36(1):1-13. doi:10.1016/S0254-6272(16)30001-2 33. Liu L, Huang Q-M, Liu Q-G, et al. Effectiveness of Dry Needling for Myofascial Trigger Points Associated With Neck and Shoulder Pain: A Systematic Review and Meta-Analysis. Arch Phys Med Rehabil. 2015;96(5):944-955. doi:10.1016/j.apmr.2014.12.015 34. Ong J, Claydon LS. The effect of dry needling for myofascial trigger points in the neck and shoulders: A systematic review and meta-analysis. J Bodyw Mov Ther. 2014;18(3):390-398. doi:10.1016/j.jbmt.2013.11.009 35. Tough EA, White AR, Cummings TM, et al. Acupuncture and dry needling in the management of myofascial trigger point pain: a systematic review and meta-analysis of randomised controlled trials. Eur J Pain. 2009;13(1):3-10. doi:10.1016/j.ejpain.2008.02.006 36. Furlan AD, Van Tulder M, Cherkin D, et al. Acupuncture and Dry-Needling for Low Back Pain: An Updated Systematic Review Within the Framework of the Cochrane Collaboration. Spine (Phila Pa 1976). 2005;30(8):944-963. 37. Lundh A, Jrgensen AW, Bero L. Association between personal conflicts of interest and recommendations on medical interventions. In: Lundh A, ed. Cochrane Database of Systematic Reviews. Chichester, UK: John Wiley & Sons, Ltd; 2013:1-7. doi:10.1002/14651858.MR000040 38. Grundy Q, Bero L, Malone R. Interactions between Non-Physician Clinicians and Industry: A Systematic Review. PLoS Med. 2013;10(11). doi:10.1371/journal.pmed.1001561 39. Hakoum MB, Anouti S, Al-Gibbawi M, et al. Reporting of financial and non-financial conflicts of interest by authors of systematic reviews: a methodological survey. BMJ Open. 2016;6(8):e011997. doi:10.1136/bmjopen-2016-011997 40. Hakoum MB, Jouni N, Abou-Jaoude EA, et al. Authors of Clinical Trials Reported Individual and Financial Conflicts of Interest More Frequently than Institutional and non-financial Ones: a Methodological Survey. J Clin Epidemiol. 2017;0(0):1-9. doi:10.1016/j.jclinepi.2017.04.002 41. Shawwa K, Kallas R, Koujanian S, et al. Requirements of clinical journals for authors disclosure of financial and non-financial conflicts of interest: A cross sectional study. PLoS One. 2016;11(3):1-12. doi:10.1371/journal.pone.0152301 42. IOM (Institute of Medicine). ClinicalPractice Guidelines We Can Trust. Washington, D.C.: National Academies Press; 2011. http://www.nap.edu/catalog.php?record_id=13058. Accessed July 23, 2017. 43. Schnemann HJ, Osborne M, Moss J, et al. An official American thoracic society policy statement: Managing conflict of interest in professional societies. Am J Respir Crit Care Med. 2009;180(6):564580. doi:10.1164/rccm.200901-0126ST 44. Munder T, Flckiger C, Gerger H, Wampold BE, Barth J. Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? A meta-analysis. J Couns Psychol. 2012;59(4):631-637. doi:10.1037/a0029571 45. Luborsky L, Rosenthal R, Diguer L, et al. The Dodo bird verdict is alive and well - Mostly. Clin Psychol Sci Pract. 2002;9(1):2-12. doi:10.1093/clipsy/9.1.2 60 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 61 46. Luborsky LL, Singer B, Luborsky LL. Comparative Studies of Psychotherapies: Is It True That Everyone Has Won and All Must Have Prizes? Arch Gen Psychiatry. 1975;32(8):995-1008. doi:10.1001/archpsyc.1975.01760260059004 47. Rosenzweig S. Some implicit common factors in diverse methods of psychotherapy. Am J Orthopsychiatry. 1936;6(3):412-415. doi:10.1111/j.1939-0025.1936.tb05248.x 48. Wampold BE, Budge SL, Laska KM, et al. Evidence-based treatments for depression and anxiety versus treatment-as-usual: A meta-analysis of direct comparisons. Clin Psychol Rev. 2011;31(8):13041312. doi:10.1016/j.cpr.2011.07.012 49. Leykin Y, DeRubeis RJ. Allegiance in psychotherapy outcome research: Separating association from bias. Clin Psychol Sci Pract. 2009;16(1):54-65. doi:10.1111/j.1468-2850.2009.01143.x 50. McLeod B. Understanding Why Therapy Allegiance Is Linked to Clinical Outcomes. Clin psychol. 2009;16(1):69-72. 51. Ioannidis JPA. Most psychotherapies do not really work, but those that might work should be assessed in biased studies. Epidemiol Psychiatr Sci. 2016;25(5):436-438. doi:10.1017/S2045796015000888 52. Panagiotou OA, Ioannidis JPA. Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol. 2012;65(7):740-747. doi:10.1016/j.jclinepi.2012.01.008 53. Holcombe J, Talman C. Financial Conflicts of Interest: Overview, Investigator Responsibilities, and COI Rules (COI-Basic). CITI Program. https://www.citiprogram.org/members/index.cfm?pageID=125. Published 2017. 54. Bauchner H, ed. Conflict of Interest in Medicine. JAMA. 2017;317(17):1707-1803. http://jamanetwork.com/journals/jama/issue/317/17. 55. Craik RL. Retaining trust. Phys Ther. 2010;90(3):322-323. doi:10.2522/ptj.2010.90.3.322 56. Viswanathan M, Patnode CD, Berkman ND, et al. Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Scientific Resource Center under Contract No. 290-2012-0004-C). AHRQ Publication No. Rockville (MD); 2017. doi:10.23970/AHRQEPCMETHGUIDE2 57. Pellegrino E. Character Formation and the Making of Good Physicians. In: Kenny N, Shelton W, eds. Lost Virtue: Professional Character Development in Medical Education. Advances i. Oxford: Elsevier JAI; 2006:1-15. 58. Pellegrino ED. The metamorphosis of medical ethics: A 30-year retrospective. J Am Med Assoc. 1993;269(9):1158-1162. http://jama.jamanetwork.com/article.aspx?articleid=403966. 59. McCullough LB. The Ethical Concept of Medicine as a Profession: Its Origins in Modern Medical Ethics and Implications for Physicians. In: Kenny N, Shelton W, eds. Lost Virtue: Professional Character Development in Medical Education. Advances i. Oxford: Elsevier JAI; 2006:17-27. 60. Chervenak FA, McCullough LB. The moral foundation of medical lead ership: The professional virtues of the physician as fiduciary of the patient. Am J Obstet Gynecol. 2001;184(5):875-880. doi:10.1067/mob.2001.113854 61. McCullough LB. John Gregory (1724- 1773) and the invention of professional relationships in medicine. J Clin Ethics. 1997;8(1):11-21. 61 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 62 62. Orr RD, Pang N, Pellegrino ED, Siegler M. Use of the hippocratic oath: A review of twentieth century practice and a content analysis of oaths administered in medical schools in the U.S. and Canada in 1993. J Clin Ethics. 1997;8(4):377-388. 63. Jotterand F. The Hippocratic Oath and contemporary medicine: Dialectic between past ideals and present reality? J Med Philos. 2005;30(1):107-128. doi:10.1080/03605310590907084 64. Pellegrino ED. Toward a reconstruction of medical morality. Am J Bioeth. 2006;6(2):65-71. doi:10.1080/15265160500508601 65. Brody H. Conflicts of Interest. In: Klugman CM, ed. Philosophy: Medical Ethics. Farmington Hills, MI: Macmillan Interdisciplinary Handbooks; 2016:261-277. doi:10.1017/CBO9780511610332 66. Rodwin MA, Okamoto AE. Physicians conflicts of interest in Japan and the United States: Lessons for the United States. J Health Polit Policy Law. 2000;25(2):343-375. 67. Bion J. Financial and intellectual conflicts of interest: confusion and clarity. Curr Opin Crit Care. 2009;15(6):583-590. doi:10.1097/MCC.0b013e328332f53a 68. Erde EL. The Development of Bioethics in the United States. In: Garrett J, Jotterand F, Ralston D, eds. The Development of Bioethics in the United States: 115 (Philosophy and Medicine). Dordrecht: Springer; 2013:179-206. doi:10.1007/978-94-007-4011-2 69. Nager AL, Kobylecka M, Pham PK, et al. News. Stein J, Zorowitz R, Harvey R, Macko R, Winstein C, eds. Acupunct Med. 2011;12(1):118-122. doi:10.1016/j.ejpain.2008.02.006 70. Erde EL. Conflicts of interests in medicine: a philosophical and ethical morphology. In: Spece R, Shimm D, Buchanan A, eds. Conflicts of Interest in Clinical Practice and Research. New York: Oxford University Press; 1996:12-41. 71. College of Physicians and Surgeons of Ontario. Principles of Practice and Duties of Physicians. The Practice Guide: Medical Professionalism and College Policies. http://www.cpso.on.ca/PoliciesPublications/The-Practice-Guide-Medical-Professionalism-and-Col/Principles-of-Practice-and-Dutiesof-Physicians. Accessed March 17, 2018. 72. Brody H, Miller FG. The Clinician-Investigator: Unavoidable but Manageable Tension. Kennedy Inst Ethics J. 2003;13(4):329-346. doi:10.1353/ken.2004.0003 73. Miller FG, Brody H. A Critique of Clinical Equipose: Therapeutic Misconception in the Ethics of Clinical Trials. Hastings Cent Rep. 2003;33(3):19-28. doi:10.2307/3528434 74. Miller PB, Weijer C. Fiduciary obligation in clinical research. J Law, Med Ethics. 2006;34(2):424-440. doi:10.1111/j.1748-720X.2006.00049.x 75. Miller PB, Weijer C. Rehabilitating equipoise. Kennedy Inst Ethics J. 2003;13(2):93-118. doi:10.1353/ken.2003.0014 76. Shamoo AE. The Myth of Equipoise in Phase 1 Clinical Trials. Medscape J Med. 2008;10(11):254. https://www.medscape.com/viewarticle/582554_4. Accessed March 30, 2018. 77. Freedman B. Equipoise and the Ethics of Clinical Research. N Engl J Med. 1987;317(3):3-16. doi:10.1056/NEJM198707163170304 78. Guyatt G, Haynes B, Jaeschke R, et al. The philosophy of evidence-based medicine. In: Guyatt GH, Drummond R, Meade MO, Cook DJ, eds. Users Guide to the Medical LIterature: A Manual for Evidence-Based Clinical Practice. 2nd ed. New York: McGraw Hill Medical; 2008:9-16. 62 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 63 doi:10.1036/007159034X 79. London AJ. Reasonable risks in clinical research: a critique and a proposal for the Integrative Approach. Stat Med. 2006;25(17):2869-2885. doi:10.1002/sim.2634 80. London AJ. Clinical Equipoise: Foundational Requirement or Fundamental Error? In: Steinbock B, ed. The Oxford Handbook of Bioethics. Oxford, UK: Oxford University Press; 2007:571-596. 81. Miller FG, Rosenstein DL, DeRenzo EG. Professional integrity in clinical research. J Am Med Assoc. 1998;280(16):1449-1454. doi:10.1001/jama.280.16.1449 82. Hey SP, London AJ, Weijer C, Rid A, Miller F. Is the concept of clinical equipoise still relevant to research? BMJ. 2017;359(December):1-4. doi:10.1136/bmj.j5787 83. Wendler D, Miller FG. Assessing research risks systematically: The net risks test. J Med Ethics. 2007;33(8):481-486. doi:10.1136/jme.2005.014043 84. Rid A, Wendler D. A Framework for Risk-Benefit Evaluations in Biomedical Research. Kennedy Inst Ethics J. 2011;21(2):141-179. 85. Litton P, Miller FG. A Normative Justification for Distinguishing the Ethics of Clinical Research from the Ethics of Medical Care. J Law, Med Ethics. 2005;33(3):566-574. doi:10.1111/j.1748720X.2005.tb00519.x 86. Miller FG, Brody H. Clinical Equipoise and the Incoherence of Research Ethics. J Med Philos. 2007;32(2):151-165. doi:10.1080/03605310701255750 87. Amiri AR, Kanesalingam K, Cro S, Casey ATH. Does source of funding and conflict of interest influence the outcome and quality of spinal research? Spine J. 2014;14(2):308-314. doi:10.1016/j.spinee.2013.10.047 88. Lundh A, Krogsbll LT, Gtzsche PC, et al. Sponsors participation in conduct and reporting of industry trials: a descriptive study. Trials. 2012;13(1):146. doi:10.1186/1745-6215-13-146 89. IOM (Institute of Medicine). Conflict of Interest and Medical Innovation. Washington, D.C.: National Academies Press; 2014. doi:10.17226/18723 90. Bero L. Addressing Bias and Conflict of Interest Among Biomedical Researchers. JAMA. 2017;317(17):1723-1724. doi:10.1001/jama.2017.3854 91. Bero LA, Grundy Q. Why Having a (Nonfinancial) Interest Is Not a Conflict of Interest. PLoS Biol. 2016;14(12):1-8. doi:10.1371/journal.pbio.2001221 92. Wiersma M, Kerridge I, Lipworth W. Dangers of neglecting non-financial conflicts of interest in health and medicine. J Med Ethics. 2017:medethics-2017-104530. doi:10.1136/medethics-2017-104530 93. Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: What it is and what it isnt - Its about integrating individual clinical expertise and the best external evidence. Br Med J. 1996;312(1):71-72. doi:10.2307/29730277 94. Larkin I, Loewenstein G. Business ModelRelated Conflict of Interests in Medicine: Problems and Potential Solutions. JAMA. 2017;317(17):1745-1746. doi:10.1001/jama.2017.2275 95. Chou R, Fu R, Carrino JA, Deyo RA. Imaging strategies for low-back pain: systematic review and meta-analysis. Lancet (London, England). 2009;373(9662):463-472. doi:10.1016/S01406736(09)60172-0 63 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 64 96. Jarvik JG, Gold LS, Comstock BA, et al. Association of early imaging for back pain with clinical outcomes in older adults. JAMA - J Am Med Assoc. 2015;313(11):1143-1153. doi:10.1001/jama.2015.1871 97. Kerry S, Hilton S, Dundas D, Rink E, Oakeshott P. Radiography for low back pain: a randomised controlled trial and observational study in primary care. Br J Gen Pract J R Coll Gen Pract. 2002;52(479):469-474. 98. Graves JM, Fulton-Kehoe D, Jarvik JG, Franklin GM. Health care utilization and costs associated with adherence to clinical practice guidelines for early magnetic resonance imaging among workers with acute occupational low back pain. Health Serv Res. 2014;49(2):645-665. doi:10.1111/14756773.12098 99. Fuhrmans V. A Novel Plan Helps Hospital Wean Itself Off Pricey Tests. The Wall Street Journal Online. 2007:1-5. 100. Moore DA, Loewenstein G. Self-interest, atomaticity, and the psychology of conflict of interest. Soc Justice Res. 2004;17(2):189-202. doi:10.1023/B:SORE.0000027409.88372.b4 101. Dana J, Loewenstein G. A Social Science Perspective on Gifts. J Am Med Assoc. 2003;290(2):252-255. 102. Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials. 2005;2(3):218-232. doi:10.1191/1740774505cn085oa 103. Vartanian LR, Schwartz MB, Brownell KD. Effects of Soft Drink Consumption on Nutrition and Health: A Systematic Review and Meta-Analysis. Am J Public Health. 2007;97(4):667-675. doi:10.2105/AJPH.2005.083782 104. Sibbald B. Death but one unintended consequence of gene-therapy trial. C Can Med Assoc J. 2001;164(11):1612. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC81135/. 105. Kearns C, Schmidt L, Glantz S. Sugar industry and coronary heart disease research: A historical analysis of internal industry documents. JAMA Intern Med. 2016;176(11):1680-1685. http://dx.doi.org/10.1001/jamainternmed.2016.5394. 106. Schillinger D, Tran J, Mangurian C, Kearns C. Do sugar-sweetened beverages cause obesity and diabetes? Industry and the manufacture of scientific controversy. Ann Intern Med. 2016;165(12):895897. doi:10.7326/L16-0534 107. McCoy MS, Emanuel EJ. Why there are no potential conflicts of interest. JAMA. 2017;317(17):1721-1722. doi:10.1001/jama.2017.2308 108. Domonoske C. 50 Years Ago, Sugar Industry Quietly Paid Scientists To Point Blame At Fat. National Public Radio:The Two-Way. http://www.npr.org/sections/thetwo-way/2016/09/13/493739074/50years-ago-sugar-industry-quietly-paid-scientists-to-point-blame-at-fat. Published September 13, 2016. Accessed October 30, 2017. 109. OConnor A. How the Sugar Industry Shifted Blame to Fat. New York Times. https://www.nytimes.com/2016/09/13/well/eat/how-the-sugar-industry-shifted-blame-tofat.html?action=click&contentCollection=Opinion&module=Trending&version=Full®ion=Margin alia&pgtype=article. Published September 12, 2016. 110. OConnor A. Studies Linked to Soda Industry Mask Health Risks. New York Times. https://www.nytimes.com/2016/11/01/well/eat/studies-linked-to-soda-industry-mask-health-risks.html. Published October 31, 2016. 64 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 65 111. Terhune C, Levey NM, Poindexter S. Database shows $3.5 billion in industry ties to doctors, hospitals. Los Angeles Times. http://www.latimes.com/business/la-fi-doctors-payments-20141001-story.html. Published 2014. Accessed January 31, 2018. 112. Grant K. The pressure of Big Pharma: Financial conflicts of interest common on medical guideline panels -. The Globe and Mail. https://www.theglobeandmail.com/news/national/the-pressure-of-bigpharma-financial-conflicts-of-interest-common-on-medical-guidelinepanels/article35389639/. Published November 12, 2017. Accessed January 31, 2017. 113. Ramshaw E. Payments to Doctors by Pharmaceutical Companies Raise Issues of Conflicts. New York Times. http://www.nytimes.com/2011/11/25/us/payments-to-doctors-by-pharmaceutical-companiesraise-issues-of-conflicts.html. Published November 24, 2011. 114. Ornstein C. From Twitter To Treatment Guidelines, Industry Influence Permeates Medicine. National Public Radio. https://www.npr.org/sections/health-shots/2017/01/17/510226214/from-twitter-totreatment-guidelines-industry-influence-permeates-medicine. Published January 17, 2017. 115. Heath D. Trial to open in deaths of cancer patients at Hutch. Seattle Times. http://community.seattletimes.nwsource.com/archive/?date=20040201&slug=hutch01. Published February 1, 2002. 116. Lopez L. Novo Nordisk has been accused of engaging in a white coat marketing scheme to sell diabetes drugs. Business Insider. http://www.businessinsider.com/novo-nordisk-diabetes-drug-lawsuit2017-4. Published April 28, 2017. 117. Silverman E. Are nurses the new sales reps? Why pharma should be worried. STAT+. November 28, 2017:1-4. 118. Kassirer JP. On the Take: How Medicines Complicity with Big Business Can Endanger Your Health. New York, New York: Oxford University Press; 2004. 119. Angell M. The Truth About the Drug Companies: How The Deceive Us and What to Do About It. Toronto, Ontario: Random House; 2004. 120. Elgin B, Bloomfield D, Chen C. When the Patient Is a Gold Mine: The Trouble With Rare-Disease Drugs. Bloom Businessweek. May 2017. https://www.bloomberg.com/news/features/2017-05-24/whenthe-patient-is-a-gold-mine-the-trouble-with-rare-disease-drugs. 121. McKinney RE, Pierce HH. Strategies for Addressing a Broader Definition of Conflicts of Interest. JAMA. 2017;20001(17):1727-1728. doi:10.1001/jama.2017.3857 122. Wadmann S. Physician-industry collaboration: Conflicts of interest and the imputation of motive. Soc Stud Sci. 2014;44(4):531-554. doi:10.1177/0306312714525678 123. Gaffan EA, Tsaousis J, Kemp-Wheeler SM. Researcher allegiance and meta-analysis: The case of cognitive therapy for depression. J Consult Clin Psychol. 1995;63(6):966-980. doi:10.1037/0022006X.63.6.966 124. Gallo SA, Lemaster M, Glisson SR. Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection. Sci Eng Ethics. 2016;22(1):189-197. doi:10.1007/s11948-015-9631-7 125. Guyatt GH, Akl EA, Hirsch J et al. The Vexing Problem of Guidelines and Conflict of Interest: A Potential Solution. Ann Intern Med. 2010;152(11):738-741. http://annals.org/aim/article/745809/vexing-problem-guidelines-conflict-interest-potential-solution. 65 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 66 126. Irwin RS. The role of conflict of interest in reporting of scientific information. Chest. 2009;136(1):253259. doi:10.1378/chest.09-0890 127. Holroyd KA, Powers SW, Andrasik F. Focus on Behavioral Methodological Issues in Clinical Trials of Drug and Behavior Therapies. Headache. 2005;45:487-492. 128. Malay DS. Financial and Nonfinancial (Allegiance) Conflicts of Interest in Clinical Research. J Foot Ankle Surg. 2016;55(6):1129. doi:10.1053/j.jfas.2016.09.004 129. Marshall E. When does intellectual passion become conflict of interest? Science (80- ). 1992;257(5070):620-623. doi:10.1126/science.1496373 130. PLoS Medicine Editors. Making sense of non-financial competing interests. PLoS Med. 2008;5(9):e199. doi:10.1371/journal.pmed.0050199 131. Poole D. When does a point of view become a conflict of interest? Crit Care Med. 2008;36(5):1688. 132. Sniderman AD, Furberg CD. Pluralism of viewpoints as the antidote to intellectual conflict of interest in guidelines. J Clin Epidemiol. 2012;65(7):705-707. doi:10.1016/j.jclinepi.2012.01.009 133. Sox HC. Conflict of Interest in Practice Guidelines Panels. JAMA. 2017;317(17):1739-1740. doi:10.1001/jama.2017.2701 134. Stead WW. The Complex and Multifaceted Aspects of Conflicts of Interest. JAMA. 2017;317(17):1765-1767. doi:10.1001/jama.2017.3435 135. Cook C, Sheets C. Clinical equipoise and personal equipoise: two necessary ingredients for reducing bias in manual therapy trials. J Man Manip Ther. 2011;19(1):55-57. doi:10.1179/106698111X12899036752014 136. Easley TJ. Medical journals, publishers, and conflict of interest. JAMA. 2017;317(17):1759-1760. doi:10.1001/jama.2015.12867 137. Fontanarosa P, Bauchner H. Conflict of interest and medical journals. JAMA. 2017;317(17):1768-1771. 138. Gottlieb JD, Bressler NM. How Should Journals Handle the Conflict of Interest of Their Editors ? Who Watches the Watchers ? JAMA. 2017;317:1757-1758. doi:10.1001/jama.2017.2207 139. OMalley A S, Pham HH, Reschovsky JD. Predictors of the growing influence of clinical practice guidelines. J Gen Intern Med. 2007;22(6):742-748. doi:10.1007/s11606-007-0155-y 140. Cheng JS, Lee MJ, Massicotte E, et al. Clinical Guidelines and Payer Policies on Fusion for the Treatment of Chronic Low Back Pain. Spine (Phila Pa 1976). 2011;36(21):S144-S163. doi:10.1097/BRS.0b013e31822ef5b4 141. Norris SL, Holmer HK, Burda BU, Ogden LA, Fu R. Conflict of interest policies for organizations producing a large number of clinical practice guidelines. PLoS One. 2012;7(5). doi:10.1371/journal.pone.0037413 142. Alexander PE, Bero L, Montori VM, et al. World Health Organization recommendations are often strong based on low confidence in effect estimates. J Clin Epidemiol. 2014;67(6):629-634. doi:10.1016/j.jclinepi.2013.09.020 143. Alexander PE, Gionfriddo MR, Li SA, et al. A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance. J Clin Epidemiol. 2016;70:111-122. doi:10.1016/j.jclinepi.2015.09.006 66 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 67 144. Avraham R. Clinical Practice Guidelines: The Warped Incentives in the U.S. Healthcare System. Am J Law Med. 2011;37:7-40. http://ssrn.com/abstract=1593133. Accessed December 1, 2017. 145. Emanuel EJ, Fuchs VR. The Perfect Storm of Overutilization Overutilization of Health Care. JAMA. 2008;299(23):2789-2791. doi:10.1001/jama.299.23.2789 146. Kirsch M. Conflicts of interest and conflicting views on physician compensation: Fee-for-service versus salaried medicine. Clin Gastroenterol Hepatol. 2010;8(8):666-668. doi:10.1016/j.cgh.2010.05.026 147. Keyhani S, Falk R, Howell EA, Bishop T, Korenstein D. Overuse and Systems of Care. Med Care. 2013;51(6):503-508. doi:10.1097/MLR.0b013e31828dbafe 148. Kale MS, Bishop TF, Federman AD, Keyhani S. Trends in the overuse of ambulatory health care services in the United States. JAMA Intern Med. 2013;173(2):142-148. doi:10.1001/2013.jamainternmed.1022 149. Begg CB, Berlin JA. Publication bias and dissemination of clinical research. J Natl Cancer Inst. 1989;81(2):107-115. 150. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Hopewell S, ed. Cochrane Database Syst Rev. 2009;(1). doi:10.1002/14651858.MR000006.pub3 151. Felson DT, Glantz L. A surplus of positive trials: weighing biases and reconsidering equipoise. Arthritis Res Ther. 2004;6(3):117-119. doi:10.1186/ar1189 152. Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993;50. 153. Fried C. Medical Experimentation: Personal Integrity and Social Policy. New. (Miller FG, Wertheimer A, eds.). Oxford, UK; 2016. 154. Hey SP, Weijer C. What questions can a placebo answer? Monash Bioeth Rev. 2016;34(1):23-36. doi:10.1007/s40592-016-0057-z 155. Van Der Graaf R, Van Delden JJM. Equipoise should be amended, not abandoned. Clin Trials. 2011;8(4):408-416. doi:10.1177/1740774511409600 156. Menke JM. Do Manual Therapies Help Low Back Pain? Spine (Phila Pa 1976). 39(7):463-472. doi:10.1097/BRS.0000000000000230 157. Van Tulder MW, Koes B, Malmivaara A. Outcome of non-invasive treatment modalities on back pain: An evidence-based review. Eur Spine J. 2006;15(SUPPL. 1):S64-S81. doi:10.1007/s00586-005-1048-6 158. Artus M, van der Windt DA, Jordan KP, Hay EM. Low back pain symptoms show a similar pattern of improvement following a wide range of primary care treatments: A systematic review of randomized clinical trials. Rheumatology. 2010;49(12):2346-2356. doi:10.1093/rheumatology/keq245 159. Bron C, de Gast A, Dommerholt J, Stegenga B, Wensing M, Oostendorp RAB. Treatment of myofascial trigger points in patients with chronic shoulder pain: a randomized, controlled trial. BMC Med. 2011;9(1):8. doi:10.1186/1741-7015-9-8 160. Page M, Green S, Kramer S, et al. Manual therapy and exercise for adhesive capsulitis. Buchbinder R, ed. Cochrane Database Syst Rev. 2014;(8):1-225. doi:10.1002/14651858.CD011275.www.cochranelibrary.com 67 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 68 161. Rzewuska M, Ferreira M, McLachlan AJ, Machado GC, Maher CG. The efficacy of conservative treatment of osteoporotic compression fractures on acute pain relief: a systematic review with metaanalysis. Eur Spine J. 2015;24(4):702-714. doi:10.1007/s00586-015-3821-5 162. Cook C, Learman K, Showalter C, Kabbaz V, OHalloran B. Early use of thrust manipulation versus non-thrust manipulation: A randomized clinical trial. Man Ther. 2013;18(3):191-198. doi:10.1016/j.math.2012.08.005 163. Bishop MD, Bialosky JE, Penza CW, Beneciuk JM, Alappattu MJ. The influence of clinical equipoise and patient preferences on outcomes of conservative manual interventions for spinal pain: an experimental study. J Pain Res. 2017;10:965-972. doi:10.2147/JPR.S130931 164. Airaksinen O, Brox JI, Cedraschi C, et al. Chapter 4: European guidelines for the management of chronic nonspecific low back pain. Eur Spine J. 2006;15(SUPPL. 2):192-300. doi:10.1007/s00586006-1072-1 165. Andersson GBJ. Epidemiological features of chronic low-back pain. 1999;354:581-585. 166. Laerum E, Brox J, Storheim K. Korsryggsmerter- Med Og Uten Nerverotaffeksjon. Oslo; 2007. 167. Maitland G. Vertebral Manipulation. 5th ed. Oxford, UK: Butterworth-Heinemann; 1997. 168. Flynn T, Fritz J, Whitman J, et al. A Clinical Prediction Rule for Classifying Patients with Low Back Pain Who Demonstrate Short-Term Improvement With Spinal Manipulation. Spine (Phila Pa 1976). 2002;27(24). 169. Cibulka MT, Delitto a, Koldehoff RM. Changes in innominate tilt after manipulation of the sacroiliac joint in patients with low back pain. An experimental study. Phys Ther. 1988;68(9):1359-1363. doi:10.1093/ptj/68.9.1359 170. White A, Editorial Board of Acupuncture in Medicine. Western medical acupuncture: a definition. Acupunct Med. 2009;27(1):33-35. doi:10.1136/aim.2008.000372 171. White A. Western medical acupuncture: a definition. Acupunct Med. 2009;27(1):33-35. doi:10.1136/aim.2008.000372 172. Rickards LD. Therapeutic needling in osteopathic practice: An evidence-informed perspective. Int J Osteopath Med. 2009;12(1):2-13. doi:10.1016/j.ijosm.2009.01.003 173. Zhou K, Ma Y, Brogan MS. Dry needling versus acupuncture: the ongoing debate. Acupunct Med. 2015;33(6):485-490. doi:10.1136/acupmed-2015-010911 174. Fan AY, Xu J, Li Y ming. Evidence and expert opinions: Dry needling versus acupuncture (II): The American Alliance for Professional Acupuncture Safety (AAPAS) White Paper 2016. Chin J Integr Med. 2017;23(2):83-90. doi:10.1007/s11655-017-2800-6 175. Liu L, Skinner M, Mcdonough SM, Kannan P, Baxter GD. STRICTA: is it time to do more? BMC Complement Altern Med. 2015;15(190):1-11. doi:10.1186/s12906-015-0714-4 176. Jia P, Tang L, Yu J, Liu J, Kang D, Sun X. The quality of reporting in randomized controlled trials of acupuncture for knee osteoarthritis: A cross-sectional survey. PLoS One. 2018;13(4):1-14. doi:10.1371/journal.pone.0195652 177. Venere K, Ridgeway K. Trigger point dry needling: the data do not support broad applicability or robust effect. J Man Manip Ther. 2016;24(1):2-4. doi:10.1080/10669817.2015.1106820 68 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 69 178. Ware JW. Misreport of trigger point diagnosis reliability. J Orthop Sport Phys Ther. 2015;45(5):426427. doi:10.2519/jospt.2015.0203 179. Unverzagt C, Berglund K, Thomas JJ. Dry Needling for Myofascial Trigger Point Pain: a Clinical Commentary. Int J Sports Phys Ther. 2015;10(3):402-418. 180. Llamas-Ramos R, Pecos-Martn D, Gallego-Izquierdo T, et al. Comparison of the short-term outcomes between trigger point dry needling and trigger point manual therapy for the management of chronic mechanical neck pain: a randomized clinical trial. J Orthop Sports Phys Ther. 2014;44(11):852-861. http://dx.doi.org/10.2519/jospt.2014.5229. 181. Mejuto-Vzquez MJ, Salom-Moreno J, Ortega-Santiago R, Truyols-Domnguez S, Fernndez-De-Laspeas C. Short- Term changes in neck pain, widespread pressure pain sensitivity, and cervical range of motion after the application of trigger point dry needling in patients with acute mechanical neck pain: A randomized clinical trial. J Orthop Sports Phys Ther. 2014;44(4):252-260. http://dx.doi.org/10.2519/jospt.2014.5108. 182. Caramagno J, Adrian L, Mueller L, Purl J. Analysis of Competencies for Dry Needling by Physical Therapists: Final Report. Alexandria, Virginia; 2015. https://docs.wixstatic.com/ugd/802f2e_5ae0c90f45964244ba2f234e734980ce.pdf. Accessed July 25, 2018. 183. Mississippi Dry Needling. https://www.msdryneedling.com/dryneedling. Published 2018. Accessed July 25, 2018. 184. Dommerholt J. Myofascial pain syndrome -- trigger points. J Musculoskelet Pain. 2009;17(3):309-318. 185. Dommerholt J, Bron C, Franssen J. Myofascial trigger points: an evidence-informed review. J Man Manip Ther (Journal Man Manip Ther. 2006;14(4):203-221. 186. Dommerholt J. Dry needling peripheral and central considerations. J Man Manip Ther. 2011;19(4):223-227. doi:10.1179/106698111X13129729552065 187. Simons DG, Dommerholt J. Literature reviews. Myofascial pain syndromes- trigger points. J Musculoskelet Pain. 2004;12(2):69-84. 188. McPartland J, Simons DG. Myofascial trigger points: translating molecular theory into manual therapy. J Man Manip Ther (Journal Man Manip Ther. 2006;14(4):232-239. 189. Chan SHH. What is being stimulated in acupuncture: Evaluation of the existence of a specific substrate. Neurosci Biobehav Rev. 1984;8(1):25-33. doi:10.1016/0149-7634(84)90018-6 190. Dunning J, Butts R, Mourad F, Young I, Flannagan S, Perreault T. Dry needling: a literature review with implications for clinical practice guidelines. Phys Ther Rev. 2014;19(4):252-265. doi:10.1179/108331913X13844245102034 191. Price S, Long AF, Godfrey M, Thomas KJ. Getting inside acupuncture trials--exploring intervention theory and rationale. BMC Complement Altern Med. 2011;11(1):22. doi:10.1186/1472-6882-11-22 192. Han J-S. Acupuncture: neuropeptide release produced by electrical stimulation of different frequencies. Trends Neurosci. 2003;26(1):17-22. doi:10.1016/S0166-2236(02)00006-1 193. Han J, Terenius L. Neurochemical basis of acupuncture analgesia. Ann Rev Pharmacol Toxicol. 1982;22:193-220. 194. Langevin HM, Bouffard NA, Badger GJ, Churchill DL, Howe AK. Subcutaneous tissue fibroblast 69 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 70 cytoskeletal remodeling induced by acupuncture: evidence for a mechanotransduction-based mechanism. J Cell Physiol. 2006;207(3):767-774. doi:10.1002/jcp.20623 195. Lin J-G, Chen W-L. Acupuncture Analgesia: A Review of Its Mechanisms of Actions. Am J Chin Med. 2008;36(4):635-645. doi:Doi 10.1142/S0192415x08006107 196. Longhurst JC. Defining Meridians: A Modern Basis of Understanding. J Acupunct Meridian Stud. 2010;3(2):67-74. doi:10.1016/S2005-2901(10)60014-3 197. Moffet HH. How might acupuncture work? A systematic review of physiologic rationales from clinical trials. BMC Complement Altern Med. 2006;6(1):25. doi:10.1186/1472-6882-6-25 198. Zhao Z-Q. Neural mechanism underlying acupuncture analgesia. Prog Neurobiol. 2008;85(4):355-375. doi:10.1016/j.pneurobio.2008.05.004 199. Zhou W. Afferent mechanisms underlying stimulation modality-related modulation of acupuncturerelated cardiovascular responses. J Appl Physiol. 2004;98(3):872-880. doi:10.1152/japplphysiol.01079.2004 200. American Physical Therapy Association. Physical Therapists & The Performance of Dry Needling: An Educational Resource.; 2012. 201. White A. Western medical acupuncture: a definition. Acupunct Med. 2009;27(1):33-35. doi:10.1136/aim.2008.000372 202. Lu G-D, Needham J. A scientific basis for acupuncture? Sciences (New York). 1979;May/June:6-10. 203. Villarreal Santiago M, Tumilty S, Mcznik A, Mani R. Does Acupuncture Alter Pain-related Functional Connectivity of the Central Nervous System? A Systematic Review. J Acupunct Meridian Stud. 2016;9(4):167-177. doi:10.1016/j.jams.2015.11.038 204. White A, Cummings M, Barlas P, et al. Defining an adequate dose of acupuncture using a neurophysiological approach a narrative review of the literature. Acupunct Med. 2008;26(2):111120. doi:10.1136/aim.26.2.111 205. Bishop MD, Torres-Cueco R, Gay CW, Lluch-Girbs E, Beneciuk JM, Bialosky JE. What effect can manual therapy have on a patients pain experience? Pain Manag. 2015;5(6):455-464. doi:10.2217/pmt.15.39 206. Bialosky JE, Bishop MD, Penza CW. Placebo Mechanisms of Manual Therapy: A Sheep in Wolfs Clothing? J Orthop Sport Phys Ther. 2017;47(5):301-304. doi:10.2519/jospt.2017.0604 207. Hawk C, Long CR, Rowell RM, Gudavalli MR, Jedlicka J. A randomized trial investigating a chiropractic manual placebo: A novel design using standardized forces in the delivery of active and control treatments. J Altern Complement Med. 2005;11(1):109-117. http://dx.doi.org/10.1089/acm.2005.11.109. 208. Autret A, Valade D, Debiais S. Placebo and other psychological interactions in headache treatment. J Headache Pain. 2012;13(3):191-198. doi:10.1007/s10194-012-0422-0 209. Southerst D, Marchand A-A, Ct P, et al. The Effectiveness of Noninvasive Interventions for Musculoskeletal Thoracic Spine and Chest Wall Pain: A Systematic Review by the Ontario Protocol for Traffic Injury Management (OPTIMa) Collaboration. J Manipulative Physiol Ther. 2015;38(7):521-531. doi:10.1016/j.jmpt.2015.06.001 210. Kong J, Fufa DT, Gerber AJ, et al. Psychophysical outcomes from a randomized pilot study of manual, 70 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 71 electro, and sham acupuncture treatment on experimentally induced thermal pain. J Pain. 2005;6(1):55-64. http://dx.doi.org/10.1016/j.jpain.2004.10.005. 211. White P, Bishop FL, Prescott P, Scott C, Little P, Lewith G. Practice, practitioner, or placebo? A multifactorial, mixed-methods randomized controlled trial of acupuncture. Pain. 2012;153(2):455-462. http://dx.doi.org/10.1016/j.pain.2011.11.007. 212. Pariente J, White P, Frackowiak RSJ, Lewith G. Expectancy and belief modulate the neuronal substrates of pain treated by acupuncture. Neuroimage. 2005;25(4):1161-1167. doi:10.1016/j.neuroimage.2005.01.016 213. Chou R, Loeser JD, Owens DK, et al. Interventional therapies, surgery, and interdisciplinary rehabilitation for low back pain: an evidence-based clinical practice guideline from the American Pain Society. Spine (Phila Pa 1976). 2009;34(10):1066-1077. doi:10.1097/BRS.0b013e3181a1390d 214. Gonzalez-Perez LM, Infante-Cossio P, Granados-Nuez M, Urresti-Lopez FJ. Treatment of temporomandibular myofascial pain with deep dry needling. Med Oral Patol Oral Cir Bucal. 2012;17(5):781-786. doi:10.4317/medoral.17822 215. Huang Y-T, Lin S-Y, Neoh C-A, Wang K-Y, Jean Y-H, Shi H-Y. Dry needling for myofascial pain: prognostic factors. J Altern Complement Med. 2011;17(8):755-762. doi:10.1089/acm.2010.0374 216. Rodrguez-Mansilla J, Gonzlez-Snchez B, De Toro Garca , et al. Effectiveness of dry needling on reducing pain intensity in patients with myofascial pain syndrome: a Meta-analysis. J Tradit Chinese Med. 2016;36(1):1-13. doi:10.1016/S0254-6272(16)30001-2 217. Search Strategy Used to Create the Systematic Reviews Subset on PubMed. https://www.nlm.nih.gov/bsd/pubmed_subsets/sysreviews_strategy.html. Published 2017. Accessed June 1, 2018. 218. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):1-10. doi:10.1186/s13643-016-0384-4 219. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed.1000097 220. PEDro Scale. PEDro Physiotherapy Evidence Database. https://www.pedro.org.au/english/downloads/pedro-scale/. Published 1999. Accessed February 1, 2020. 221. Ware JW. The influence of conflict of interest on reporting of outcomes and conclusions in clinical trials of needling therapies: A narrative review. Unpubl Manuscr. 2018. 222. Baerlocher MO, Newton M, Gautam T, Tomlinson G, Detsky AS. The meaning of author order in medical research. J Investig Med. 2007;55(4):174-180. doi:10.2310/6650.2007.06044 223. Falkenstrom F, Markowitz JC, Jonker H, Philips B, Holmqvist R. Can Psychotherapists Function as Their Own Controls? Meta- Analysis of the Crossed Therapist Design in Comparative Psychotherapy Trials. J Clin Psychiatry. 2013;74(5):482-491. doi:10.4088/JCP.12r07848.Can 224. Sterling M, Valentin S, Vicenzino B, Souvlis T, Connelly LB. Dry needling and exercise for chronic whiplash - a randomised controlled trial. BMC Musculoskelet Disord. 2009;10:160. doi:10.1186/14712474-10-160 225. Fernndez-Carnero J, La Touche R, Ortega-Santiago R, et al. Short-term effects of dry needling of active myofascial trigger points in the masseter muscle in patients with temporomandibular disorders. J 71 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 72 Orofac Pain. 2010;24(1):106-112. 226. Zheng Y, Shi D, Wu X, et al. Ultrasound-guided miniscalpel-needle release versus dry needling for chronic neck pain: a randomized controlled trial. Evidence-Based Complement Altern Med. 2014;235817:235817. 227. Santos RB da C, Carneiro MIS, Oliveira DM de, Maciel AB do R, Monte-Silva KK do, Arajo M das GR. Impact of dry needling and ischemic pressure in the myofascial syndrome: controlled clinical trial. Fisioter em Mov. 2014;27(4):515-522. doi:10.1590/0103-5150.027.004.AO03 228. Chou L-W, Hsieh Y-L, Kao M-J, Hong C-Z. Remote Influences of Acupuncture on the Pain Intensity and the Amplitude Changes of Endplate Noise in the Myofascial Trigger Point of the Upper Trapezius Muscle. Arch Phys Med Rehabil. 2009;90(6):905-912. 229. Chou LW, Hsieh YL, Chen HS, Hong CZ, Kao MJ, Han TI. Remote therapeutic effectiveness of acupuncture in treating myofascial trigger point of the upper trapezius muscle. Am J Phys Med Rehabil. 2011;90(12):1036-1049. 230. Couto C, De Souza ICC, Torres ILS, Fregni F, Caumo W. Paraspinal stimulation combined with trigger point needling and needle rotation for the treatment of myofascial pain: A randomized shamcontrolled clinical trial. Clin J Pain. 2014;30(3):214-223. doi:10.1097/AJP.0b013e3182934b8d 231. Rayegani SM, Bayat M, Bahrami MH, Raeissadat SA, Kargozar E. Comparison of dry needling and physiotherapy in treatment of myofascial pain syndrome. Clin Rheumatol. 2014;33(6):859-864. doi:10.1007/s10067-013-2448-3 232. Tough EA, White AR, Richards SH, Campbell JL. Myofascial trigger point needling for whiplash associated pain - A feasibility study. Man Ther. 2010;15(6):529-535. doi:10.1016/j.math.2010.05.010 233. Liu L, Huang Q-M, Liu Q-G, et al. Effectiveness of Dry Needling for Myofascial Trigger Points Associated With Neck and Shoulder Pain: A Systematic Review and Meta-Analysis. Arch Phys Med Rehabil. 2015;96(5):944-955. doi:10.1016/j.apmr.2014.12.015 234. Silva ROF da, Conti PCR, Araujo C dos RP, Silva R dos S. Evaluation of dry needling and 0.5% lidocaine injection therapies in myofascial pain trigger points in masticatory muscles. Dental Press J Orthod. 2012;17(2):113-118. 235. Tekin L, Akarsu S, Durmu O, akar E, Diner U, Kiralp MZ. The effect of dry needling in the treatment of myofascial pain syndrome: A randomized double-blinded placebo-controlled trial. Clin Rheumatol. 2013;32(3):309-315. http://dx.doi.org/10.1007/s10067-012-2112-3. 236. Tsai C-T, Hsieh L-F, Kuan T-S, Kao M-J, Chou L-W, Hong C-Z. Remote Effects of Dry Needling on the Irritability of the Myofascial Trigger Point in the Upper Trapezius Muscle. Am J Phys Med Rehabil. 2010;89(2):133-140. doi:10.1097/PHM.0b013e3181a5b1bc 237. Venncio R de A, Alencar Jr FGP, Zamperini C. Different Substances and Dry Needling Injections in Patients with Myofascial Pain and Headaches. J Craniomandib Sleep Pract. 2008;26(2):96-103. doi:10.1179/crn.2008.014 238. Venancio R de A, Alencar Jr. FGP, Zamperini C. Botulinum toxin, lidocaine, and dry-needling injections in patients with myofascial pain and headaches. J Craniomandib Pract. 2009;27(1):1-8. 239. Yeganeh Lari A, Okhovatian F, Naimi S sadat, Baghban AA. The effect of the combination of dry needling and MET on latent trigger point upper trapezius in females. Man Ther. 2016;21:204-209. doi:10.1016/j.math.2015.08.004 72 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 73 240. Ay S, Evcik D, Tur BS. Comparison of injection methods in myofascial pain syndrome: A randomized controlled trial. Clin Rheumatol. 2010;29(1):19-23. doi:10.1007/s10067-009-1307-8 241. Campa-Moran I, Rey-Gudin E, Fernndez-Carnero J, et al. Comparison of dry needling versus orthopedic manual therapy in patients with myofascial chronic neck pain: A single-blind, randomized pilot study. Pain Res Treat. 2015;2015. doi:10.1155/2015/327307 242. Draolu D, Vural M, Karan A, et al. Effectiveness of dry needling for the treatment of temporomandibular myofascial pain: A double-blind, randomized, placebo controlled study. J Back Musculoskelet Rehabil. 2012;25(4):285-290. doi:10.3233/BMR-2012-0338 243. Edwards J, Knowles N. Superficial Dry Needling and Active Stretching in the Treatment of Myofascial Pain - A Randomised Controlled Trial. Acupunct Med. 2003;21(3):80-86. doi:10.1136/aim.21.3.80 244. Itoh K, Saito S, Sahara S, Naitoh Y, Imai K, Kitakoji H. Randomized trial of trigger point acupuncture treatment for chronic shoulder pain: A preliminary study. J Acupunct Meridian Stud. 2014;7(2):59-64. doi:10.1016/j.jams.2013.02.002 245. Kamanli A, Kaya A, Ardicoglu O, Ozgocmen S, Zengin FO, Bayik Y. Comparison of lidocaine injection, botulinum toxin injection, and dry needling to trigger points in myofascial pain syndrome. Rheumatol Int. 2005;25(8):604-611. doi:10.1007/s00296-004-0485-6 246. Mayoral O, Salvat I, Martn MT, et al. Efficacy of myofascial trigger point dry needling in the prevention of pain after total knee arthroplasty: A randomized, double-blinded, placebo-controlled trial. Evidence-Based Complement Altern Med. 2013;2013. 247. Chou L-W, Hsieh Y-L, Chen H-S, Hong C-Z, Kao M-J, Han T-I. Remote therapeutic effectiveness of acupuncture in treating myofascial trigger point of the upper trapezius muscle. Am J Phys Med Rehabil. 2011;90(12):1036-1049. doi:10.1097/PHM.0b013e3182328875 248. Dragoo JL, Wasterlain AS, Braun HJ, Nead KT. Platelet-rich plasma as a treatment for patellar tendinopathy: A double-blind, randomized controlled trial. Am J Sports Med. 2014;42(3):610-618. doi:10.1177/0363546513518416 249. Weber MA. Academic physicians confront a hostile world: The creation of ACRE. J Clin Hypertens. 2009;11(10):533-536. doi:10.1111/j.1751-7176.2009.00192.x 250. Lichter AS. Conflict of Interest and the Integrity of the Medical Profession. JAMA. 2017;317(17):1725. doi:10.1001/jama.2017.3191 251. Bosch X, Pericas JM, Hernndez C, Doti P. Financial, nonfinancial and editors conflicts of interest in high-impact biomedical journals. Eur J Clin Invest. 2013;43(7):660-667. doi:10.1111/eci.12090 252. Lenzer J. When is a point of view a conflict of interest? BMJ. 2016;355(November):11-13. doi:10.1136/bmj.i6194 253. Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial conflicts of interest in academic grant evaluation: A qualitative study of multiple stakeholders in france. von Elm E, ed. PLoS One. 2012;7(4):e35247. doi:10.1371/journal.pone.0035247 254. Levinsky N. Nonfinancial COI in research. N Engl J Med. 2002;347(10):759-761. 255. Cuijpers P, Driessen E, Hollon SD, van Oppen P, Barth J, Andersson G. The efficacy of non-directive supportive therapy for adult depression: A meta-analysis. Clin Psychol Rev. 2012;32(4):280-291. doi:10.1016/j.cpr.2012.01.003 73 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 74 256. MacPherson H, Altman DG, Hammerschlag R, et al. Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): Extending the CONSORT statement. J Evid Based Med. 2010;3(3):140-155. doi:10.1111/j.1756-5391.2010.01086.x 257. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348. 258. Maher C, Sherrington C, Herbert R, Moseley A, Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther. 2003;83(July 2017):713-721. 259. Trinh K, Graham N, Irnich D, Cameron ID, Forget M. Acupuncture for neck disorders. In: Trinh K, ed. Cochrane Database of Systematic Reviews. Chichester, UK: John Wiley & Sons, Ltd; 2016. doi:10.1002/14651858.CD004870.pub4 260. van Tulder M, Furlan A, Bombardier C, Bouter L. Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group. Spine (Phila Pa 1976). 2003;28(12):1290-1299. doi:10.1097/BRS.0b013e3181b1c99f 261. MacDermid JC. An introduction to evidence-based practice for hand therapists. J hand Ther. 2004;17(2):105-117. doi:10.1197/j.jht.2004.02.001 262. Higgins JP, Altman DG. Cochrane Handbook of Systematic Reviews of Interventions. In: Higgins JP, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. 5.0.1. Wiley-Blackwell; 2008:188-235. 263. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials. 1996;17(1):1-12. doi:10.1016/0197-2456(95)00134-4 264. White A, Foster NE, Cummings M, Barlas P. Acupuncture treatment for chronic knee pain: a systematic review. Rheumatology. 2007;46(3):384-390. doi:10.1093/rheumatology/kel413 265. van Tulder MW, Assendelft WJ, Koes BW, Bouter LM. Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for Spinal Disorders. Spine (Phila Pa 1976). 1997;22(20):2323-2330. 266. Raman S, Moraes FY, Mendez LC, et al. The relationship of study and authorship characteristics on trial sponsorship and self-reported conflicts of interest among neuro-oncology clinical trials. J Neurooncol. 2018;139(1):195-203. doi:10.1007/s11060-018-2860-2 267. Darmon M, Helms J, De Jong A, et al. Time trends in the reporting of conflicts of interest, funding and affiliation with industry in intensive care research: a systematic review. Intensive Care Med. 2018;44(10):1669-1678. doi:10.1007/s00134-018-5350-2 268. Leite ETT, Moraes FY, Marta GN, et al. Trial sponsorship and self-reported conflicts of interest in breast cancer radiation therapy: An analysis of prospective clinical trials. Breast. 2017;33:29-33. doi:10.1016/j.breast.2017.02.016 269. Bridoux V, Moutel G, Schwarz L, Michot F, Herve C, Tuech JJ. Disclosure of funding sources and conflicts of interest in phase III surgical trials: Survey of ten general surgery journals. World J Surg. 2015;38(10):2487-2493. doi:10.1007/s00268-014-2580-5 270. Braithwaite FA, Walters JL, Li LSK, Moseley GL, Williams MT, McEvoy MP. Effectiveness and adequacy of blinding in the moderation of pain outcomes: Systematic review and meta-analyses of dry needling trials. PeerJ. 2018;6:e5318. doi:10.7717/peerj.5318 74 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 75 Table 1. Characteristics of meta-analyses of randomized clinical trials investigating effects of needling therapy on pain for musculoskeletal pain disorders Study Gattie et al, 201726 Rodriguez-Mansilla et al, 201632 Trinh et al, 2016259 Liu et al, 201533 Kietrys et al, 201325 Ong&Claydon, 201334 Tough et al, 200935 White et al, 2007264 Furlan et al, 200536 Journal Journal of Orthopaedic & Sports Physical Therapy Journal of Traditional Chinese Medicine Cochrane Library Archives of Physical Medicine and Rehabilitation Journal of Orthopaedic & Sports Physical Therapy Journal of Bodywork & Movement Therapies European Journal of Pain Rheumatology Type of NT DN DN AC* Condition(s) treated Post-op SP, chronic NP, MPS, LBP, chronic ankle sprain, WAD, TPs in neck mm NP, MPS, SP, LBP, posterior thigh pain, HA, lateral elbow pain NP, WAD, MPS, NP and HA, NP w/ radicular symptoms Follow-up Time point(s) for meta-analyses Immediate to 12 wks (short term), 6 to 12 mos (long term) DN>placebo or control immediate to 3-4 wks PEDro258 Immediate, 1d-3 mos (short term), 3 mos to 1yr, >1yr Immediate to 3d, 9-28d (medium term), 2-6 mos (long term) Immediate, approx. 4 wks compared to sham Cochrane Back Review Group260 DN superior to sham/control immediately and approx. 4 wks MacDermid261 No difference between DN and INJ PEDro258/ Cochrane Handbook262 Jadad263 Chronic MPS, TPs in neck muscles, WAD, SP DN MPS, NP, UE pain, SP DN NP due to TPs, MPS Immediate, 1-4 wks, 3-6 mos DN NP, posterior thigh pain, LBP, TPs in neck muscles Knee pain due to OA (pooled) 2-12 wks (short-term) Acute LBP, Chronic LBP DN more effective than sham/control in medium term n/a MA superior to sham PEDro258 Cochrane Back Review Group265 Cochrane Back Review Group260 Immediate, DN: immediate time point only <3mos (short term), compared to placebo 3-12mos (intermediate) Abbreviations: DN-Dry Needling, AC-acupuncture, MA- Western medical acupuncture, SP-shoulder pain, NP-neck pain, MPS-myofascial pain syndrome, LBP-low back pain, WAD-whiplash associated disorder, TPs-trigger points, HA-headache, UE-upper extremity, TMD-temporomandibular disorder, INJ-injection of various medications (e.g. anesthetics, botulinum), RHB-multidisciplinary rehabilitation, ROM-range of motion, PPT-pressure pain threshold, PEDro-Physiotherapy Evidence Database *Combined results from trials of both MA and Eastern acupuncture. **Included trials that combined both Eastern and Western theoretical frameworks of needling therapy Authors investigated MA within Western scientific framework and defined an adequate dose using strict criteria. () limited, () moderate evidence that needling therapy is more effective than control treatment Slightly modified by authors Spine AC & DN Study Quality/ Risk of Bias Assessment Tool PEDro258 Immediate, 3 to 4 wks DN** MA Reported results in immediate to short/medium term compared to sham 75 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 76 Table 2. Needling therapy companies offering courses to physical therapists Company Kinetacore Year Est. 2006 Myopain Late 1990s Spinal Manipulation Institute 2007 IAOM-US (Optimal Dry Needling Solutions) 2014 IAMPT 2011 Integrative Dry Needling Systemic Dry Needling 2009 Benchmark Rehab 2015 2015 Hands-on Seminars 2004 The Dry Needling Institute Master Dry Needling 2008 Dr. Trinhs Dry Needling Course (McMaster Univ.) Mississippi Dry Needling 2015 2017 2018 Course Title Course Fee Contact Hours $795 $1095 $1095 $1095 $1095 $1200 $1200 $900 $1295 $1295 $1295 $1295 $2000 $795 27 26 25 26 26.5 20/27 20/27 16 27 27 25 25 27 25 $795 25 Functional Dry Needling Level 1 Functional Dry Needling Level 2 Advanced Functional Dry Needling Level 3 Functional Therapeutics Functional Dry Needling of Pelvic Floor DN-1 Foundations I DN-2 Foundations II DN-3 Advanced DN-1: Dry Needling for Craniofacial, Cervicothoracic & Upper Extremity Conditions DN-2: Dry Needling for Lumbopelvic & Lower Extremity Conditions Dry Needling I Dry Needling II Dry Needling for Hand Therapists Dry Needling Pelvic Rehab Trigger Point Dry Needling Level 1 Trigger Point Dry Needling Level 2 Trigger Point Dry Needling Level 3 Foundation Dry Needling Advanced Dry Needling Foundations in Dry Needling Advanced Dry Needling Anatomical Dissection and Dry Needling DN1: Dry Needling of the Lumbar Region & Lower Quarter DN2: Dry Needling of the Cervicothoracic Region and Upper Quarter DN3: Advanced Dry Needling Dry Needling Basic Program Dry Needling Intermediate Program Dry Needling Advanced Program DN Level 1 DN Level 2 Master Dry Needling Level-1 Master Dry Needling Level-2 Dry Needling ProgramAn Evidence-Based Approach $1250 $1250 $1000 $1000/1250 $1000 $995 $995 $1095 $795 $595 $795 $795 $895 $1199 n/a $995 $995 $1495 16 24 24 24 21 21 27 27 up to 50 Course One Course Two $800 $800 54 (27 each) *2- and 3-day courses list for same fee Based on information provided online indicating course offering in Florida open to PTs (http://www.dryneedlingprogram.com/about.html). Accessed on 8/17/2017. Not offered in North America Approximate course fees converted from Euros to U.S. dollars as of 4/29/2018 Courses offered on consecutive weekends at single course fee of $1600 76 29/37* 25/33* 18 19/27 16 34 34 32 27 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 77 Table 3. General study characteristics (N=62) Funding source, n(%) Internal External^ Non-funded Not reported# Publication year, n(%) 2010 2000,<2010 <2000 Number of participants, n(%) <30 30,<60 60 Type of control group, n(%) Active Sham/Placebo Inactive No control Study quality score range, n(%) <50% 50%,<75% 75% Unclear or lacking study quality criterion, n(%)** Random allocation Allocation concealment Blinded subjects/clinicians Blinded assessors Complete data Trial registration, n(%) Reported Not reported Overall* Report CA/PA Registry 15(24) 16(26) 13(21) 18(29) 6(10) 15(24) 2(3) - 7(11) 1(2) 11(18) - 2(3) 0(0) 0(0) - 42(68) 15(24) 5(8) 16(26) 27(43) 19(31) 21(34) 21(34) 4(6) 16(26) 15(23) 33(53) 14(21) 14(23) 32(52) 58(94) 21(34) 18(29) 9(15) 53(85) Abbreviations: CA-corresponding author, PA-primary author. *Three studies had the same funding information included in both the report and online registry. The report data is included in the Table. ^One trial had both internal and external funding and was categorized as external. #Two methods were used to contact CAs/PAs for funding information: a) via email address provided in study report and, if no response, b) attempt to contact author on the ResearchGate platform (www.researchgate.net). 11(18%) of DN trials had <20 participants For studies assessed in >1 review with the same study quality tool, the lowest score was used. For studies that were assessed using >1 tool, an average score was calculated using the lowest score on each tool. **Total number of studies with quality criteria listed in report=55. An overall score only, without individual scoring of individual criteria, was provided for 5 studies that were included in 1 review,233 and no quality score was determined for 2 studies in the one review216 they appeared in. The 4 study quality criteria domains common to each tool were used to avoid validity issues. Includes Cochrane risk of bias tool, Cochrane Back Review Group tool, and Dutch Cochrane study tool One studys registry number was not valid and therefore included in the Not reported category.227 77 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 78 Table 4.General characteristics of included dry needling trials stratified by funding source Overall (N=62) Type of control Active Inactive Sham/placebo No control Publication year 2010 2000,<2010 <2000 No. of participants <30 30,<60 60 Avg. quality score <50% 50%,<75% 75% Unclear or lacking study quality criterion Random allocation Allocation concealed Blinded subj/clinicians Blinded assessors Complete data Internal (n=15) External (n=16) Non-funded (n=13) Not reported (n=18) n % n %N %n n %N %n n %N %n n %N %n 21 4 21 16 34 6 34 26 11 0 3 1 18 0 5 2 73 0 20 7 5 0 7 3 8 0 11 5 31 0 44 19 5 2 3 3 8 3 5 5 38 15 23 23 2 2 6 9 3 3 10 15 11 11 33 50 42 15 5 68 24 8 10 3 2 16 5 3 67 20 13 10 4 2 16 6 3 62 25 13 9 4 0 15 6 0 69 31 0 13 4 1 21 6 2 72 22 6 17 27 18 27 44 29 4 6 5 6 10 8 27 40 33 2 8 6 3 13 8 12 50 38 6 4 3 10 6 5 46 31 23 5 9 4 8 15 6 28 50 22 14 34 12 23 55 19 4 8 3 6 13 5 27 53 20 0 10 6 0 16 8 0 62 38 3 7 3 5 11 5 23 54 23 7 10 1 11 16 2 39 56 6 14 32 58 21 18 23 52 94 34 29 3 7 13 7 7 5 13 24 13 13 20 47 87 47 47 3 4 16 1 3 5 6 26 2 5 19 25 2 5 13 5 1 3 8 21 8 2 15 38 6 13 16 7 6 10 21 26 11 10 33 72 89 39 33 100 6 19 100 38 8 Table 5. Rating of Researcher Allegiance in Dry Needling Studies [Report (R) N=34, Survey (S) N=20] Developed/ Trained Supervised Advocated for Taught DN Provided DN Clinicians Clinicians DN Course Response R S R S R S R S R S Yes 15(75)* 14(41) 18(90) 1(3) 13(65) 1(3) 16(80) 8(24) 13(65) 0(0) n(%) No 2(6) 2(10) 7(21) 7(35) 7(21) 4(20) 26(76) 7(35) 0(0) 5(25) n(%) Not 34 0(0) 26(76) 0(0) 26(76) 0(0) 0(0) 0(0) 0(0) reported 17(50) (100) n(%) *Of the 15 survey respondents reporting teaching DN, 22 major authors in 15 studies taught academic courses and 16 major authors in 11 studies taught in continuing education courses. 78 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS Figure 1: The Health Care Professional Behavior Cycle IndustryPublic ClinicianPatient InvestigatorStudy participant Profession -Public Professional Behavior Figure 2: Causal Chain of Impaired Professional Judgment107 Primary interest at risk of bias due to secondary interest Secondary interest biases judgment 79 Biased judgment harms patient/study subject 79 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS Appendix 1. PRISMA Flow Chart Total articles found, n=903 {Systematic [sb] AND (acupuncture [MeSH Terms] OR acupuncture therapy [MeSH Terms] OR acupuncture [tiab] OR acupuncture [Text Word] OR dry needling [tiab] OR intramuscular stimulation [tiab])} (Conducted 7/16/2018) Articles remaining after duplicates removed, n=845 Retained from scoping search for title and abstract review, n=100 Retained from title and abstract review, n=64 * Total articles found, n=26 {systematic[sb] AND dry needling[ti]} (Conducted 8/24/2018) Total articles excluded, n=744 wrong population, n=549 wrong publication type, n=140 wrong intervention, n=85 wrong study design, n=77 wrong outcome, n=30 withdrawn, n=5 (Sum of excluded n>744 as articles could have more than 1 exclusion criterion, n=90) Total articles excluded, n=36 wrong intervention, n=16 wrong publication type, n=8 wrong study design, n=8 Non-English language, n=6 wrong outcome, n=3 wrong population, n=1 (Sum of excluded n>36 as articles could have more than 1 exclusion criterion, n=7) Total articles excluded, n=10 mixed acupuncture/dry needling, n=4 no citation in references section for tool used to assess study quality, n=4 DN trials not cited at Pubmed, n=2 Total articles retained for full text full text review, n=16 with 62 unique trials of DN *38 of the retained systematic reviews contained trials with a mix of NT styles; including DN, acupuncture and/or Western medical acupuncture. Therefore, a focused search was conducted with the terms above using the same time frame between 1/1/2013 and 7/16/2018 and then cross-referenced with studies from original title and abstract screening. All 26 systematic reviews were contained within the original screening procedure and 10 of these were excluded for reasons shown. 80 80 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 81 Appendix 2. COI Classifications Individual financial COI: arises from any payment, research funding, consultancy, advisory board membership, and the like from the manufacturer of a drug or device under consideration. This type of COI may involve the individuals themselves, their families or a business they own. Typically, a timeframe of three to five years is considered for these COIs. Individual professional COI: when an individual is engaged in a specific activity as ones main paid occupation; this provides no direct benefit however it can result in a COI or increased chance of bias because ones profession eventually provides a financial benefit; when related to the issue under consideration, an individual is expected to not speak against a clinical service/intervention that provides indirect benefit. Individual intellectual COI: arises from scholarly activities by the individual related to the specific issue under consideration. Institutional financial COI: arises when an institution to which an individual belongs has a relationship with the manufacturer of a drug or device under consideration. Such institutions include academic medical centers and professional societies. Institutional advocatory COI: arises when an individual (paid employee or unpaid member) belongs to an institution/organization that: has missions, objectives or strategies that include statements related to the issue under consideration; support the conduct of research to promote a concept related to the issue under consideration; has senior officials who act on behalf of the institution [that] have COI related to the specific questions of interest; has professionals who advocate for clinical services related to the specific question of interest but dont provide those services themselves. 81 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS Appendix 3. Researcher Allegiance Survey Researcher Allegiance in Clinical Trials of Dry Needling Study Identification Our sample contains reports that list the same corresponding author for multiple studies. In order that we can accurately identify the study with the survey responses, please copy and paste the study citation that was indicated on the email invitation to this survey. You must enter the study citation to continue to the survey questions. * 1. Copy and paste citation below. 82 82 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 83 Researcher Allegiance in Clinical Trials of Dry Needling Development of Dry Needling Intervention "Developed" means an author was directly responsible for the development of the particular style or implementation of dry needling used in the study. This includes formulation of the underlying conceptual model used to rationalize dry needling as used in the study. For example, if a study author has been involved in research investigating the concepts of myofascial pain syndrome and trigger points, and the participants in the study met the criteria for this diagnosis, that author would be a developer of the dry needling intervention used in the study. "Provide" means the author administered the active dry needling intervention during the study. Please choose Yes, No, or Don't know/don't recall. * 2. Did any of the authors develop or provide the dry needling intervention used in this study? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Development of Dry Needling Intervention Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." * 3. Which author(s) developed or provided the dry needling intervention used in this study? 1st author 2nd author Last author None of the above 83 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 84 Researcher Allegiance in Clinical Trials of Dry Needling Training of Clinicians Who Provided Dry Needling "Training" includes providing direct instruction of the dry needling intervention used in the study to the clinicians who provided the intervention to study participants OR conducting the training of any of the research assistants/clinicians used in the study to train those who provided the intervention to study participants. Please choose Yes, No, or Don't know/don't recall. * 4. Did any of the authors conduct the training of the clinicians that provided dry needling in the study? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Training of Clinician(s) Who Provided Dry Needling Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." * 5. Which author(s) conducted the training of the clinicians that provided dry needling in the study? 1st author 2nd author Last author None of the above 84 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 85 Researcher Allegiance in Clinical Trials of Dry Needling Supervision of Clinicians Who Provided Dry Needling "Supervise" refers to any formal interaction between the authors and the clinicians who provided the dry needling intervention used in the study. Please choose Yes, No, or Don't know/don't recall. * 6. Did any of the authors supervise the clinicians who provided the dry needling intervention used in this study? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Supervision of Clinicians Who Provided Dry Needling Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." * 7. Which author(s) supervised the clinicians who provided the dry needling intervention used in the study 1st author 2nd author Last author None of the above 85 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 86 Researcher Allegiance in Clinical Trials of Dry Needling Advocacy for Dry Needling "Advocated" for dry needling refers to an author that participated as an investigator in a previously published trial (RCT, pilot RCT, quasi-experimental study, case series, or case report) that was cited in the references section of this study. Please choose Yes, No, or Don't know/don't recall. * 8. Has any author advocated for the dry needling intervention used in the study at any time in the past? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Advocacy for Dry Needling Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." * 9. Which author(s) advocated in the past for the dry needling intervention used in the study? 1st author 2nd author Last author None of the above 86 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 87 Researcher Allegiance in Clinical Trials of Dry Needling Academic Instruction in Dry Needling "Teach" refers to providing BOTH clinical and didactic instruction in dry needling in an academic setting, including in an entry-level professional or a post-graduate residency training program. Please choose Yes, No, or Don't know/don't recall. * 10. Did any of the authors teach dry needling in an academic course prior to or during implementation of the study? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Academic Instruction in Dry Needling Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." * 11. Which author(s) taught dry needling in an academic course prior to or during implementation of the study? 1st author 2nd author Last author None of the above 87 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 88 Researcher Allegiance in Clinical Trials of Dry Needling Continuing Education in Dry Needling "Teach" refers to providing clinical and didactic instruction in dry needling as part of a continuing education course, such as a weekend certification course in dry needling. Please choose Yes, No, or Don't know/don't recall. * 12. Did any of the authors teach dry needling in a continuing education course prior to or during implementation of the study? Yes No Don't know/don't recall Researcher Allegiance in Clinical Trials of Dry Needling Continuing Education in Dry Needling Please indicate ALL that apply. If none of the author choices applies to your study, then choose "None of the above." If there were less than 3 authors, please limit choices to "1st author" and/or "2nd author." 13. Which author(s) taught dry needling in a continuing education course prior to or during implementation of the study? 1st author 2nd author Last author None of the above 88 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 89 Appendix 4. Membership in Publication Guidelines Organizations of Journals in which Dry Needling Trials were Published Membership Status # of DN Journal Title* trials ICMJE COPE WAME Acupuncture in Medicine 3 American Journal of Physical Medicine & Rehabilitation 3 + American Journal of Sports Medicine 1 + + Archives of Physical Medicine and Rehabilitation 2 + British Journal of Sports Medicine 1 + Chiropractic and Manual Therapies 1 + Clinical Journal of Pain 2 -(2018) + Clinical Rehabilitation 1 -(2017) + Clinical Rheumatology 3 Complementary Therapies in Medicine 1 + Dental Press Journal of Orthodontics 1 -(2014) Electromyography and Clinical Neurophysiology 1 Evidence-Based Complementary & Alternative Medicine 2 + Fisioterapia em Movimento 1 International J of Clinical and Experimental Medicine 1 Journal of Acupuncture and Meridian Studies 1 + Journal of Back & Musculoskeletal Rehabilitation 1 Journal of Bodywork and Movement Therapies 2 + Journal of Geriatric Physical Therapy 2 + Journal of Musculoskeletal Pain 2 + Journal of Oral Science 1 Journal of Orofacial Pain 2 Journal of Orthopaedic & Sports Physical Therapy 2 Journal of Pain 1 + Journal of Rehabilitation Medicine 2 Journal of Rehabilitation Sciences and Research 1 + Journal of Research in Medical Sciences 1 -(2018) + Manual Therapy 2 Medical journal of the Islamic Republic of Iran 1 + + Medicina Oral, Patologia Oral y Cirugia Bucal 2 Pain 2 -(2015) + Pain Research and Treatment 1 + Photomedicine and Laser Surgery 1 Physical Therapy 1 + PM&R 1 + + Rheumatology International 1 + The Journal of Alternative and Complementary Medicine 2 The Journal of Craniomandibular and Sleep Practice 2 + Turkish Journal of Rheumatology 1 Totals 59 3 21 2 Abbreviations: ICMJE=International Committee of Medical Journal Editors, COPE=Committee on Publication Ethics, WAME=World Association of Medical Editors, (+)=member, (-)=non-member *None of the three organizations offered membership prior to 1995, therefore no DN trials prior to 1995 are included (n=3). Year of journal membership listing shown in parentheses. If (-), DN trial(s) published same or prior year to membership. 89 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 90 Appendix 5. In-depth discussion on study quality and funding source Seventy-one percent of DN trials provided information on a funding source either in the report or by two methods of attempting to contact a study author. However, only 39% of DN trials reported a funding source in the published manuscript. Information on funding from the remaining studies was obtained from a CA via email. In comparison with recent data from various disciplines within clinical biomedical research, the COI disclosure/funding source reporting rates in neuro-oncology trials were 85.9%/83.1% (2010-2015),266 in breast cancer radiation trials 68%/62.5% (2004-2014),267 and in critical care 65%/41% (2001-2016).268 Also, in 657 RCTs published in ten surgical journals between 2005 and 2010, Bridoux et al269 found COI disclosure and funding source reporting rates of 25.1% and 47%, respectively. However, the COI/funding source data from these biomedical trials are based on unsolicited reporting rates from the trial reports or registry. Since the data include reporting of funding sources from 19 CAs (31% of DN trials) who were contacted by two different methods, a higher reporting rate of funding source would be expected in this study compared to those that relied only on published data in the report or online registry. An interesting finding of this work was that when study quality was stratified by funding (yes/no), funded studies were generally of higher methodological quality. At first glance, with respect to the three key domains of risk of bias,270 the data suggest the differences between funded and non-funded studies are minimal. However, given that the not reported and seven trials classified as internal are more likely non-funded, the differences in allocation concealment and blinding of assessors became evident. The number of funded trials failing to conceal allocation then is 8 (13%) compared to 21 90 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 91 (34%) non-funded trials. The number of studies that failed to blind assessors then is 6 (10%) for funded trials and 17 (27%) for non-funded trials. Moreover, similar frequencies of funded studies with 30 participants as non-funded studies with <30 participants were found, ostensibly suggesting that funded studies are more highly-powered, and therefore able to detect an effect from DN treatment if one exists. These results might either suggest that the funding of DN trials may serve as a big stick to comply with important domains of study quality or that more funds are available to ensure a high quality study. 91 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 92 Appendix 6. List of Dry Needling Trials 1. Arias-Bura JL, Fernndez-de-las-Peas C, Palacios-Cea M, Koppenhaver SL, Salom-Moreno J. Exercises and Dry Needling for Subacromial Pain Syndrome: A Randomized Parallel-Group Trial. J Pain. 2017;18(1):11-18. doi:10.1016/j.jpain.2016.08.013 2. Ay S, Evcik D, Tur BS. Comparison of injection methods in myofascial pain syndrome: A randomized controlled trial. Clin Rheumatol. 2010;29(1):19-23. doi:10.1007/s10067-009-1307-8 3. Bahadir C, Majlesi J, Unalan H. The effect of high-power pain threshold ultrasound therapy on the electrical activity of trigger points and local twitch response on electromyography: a preliminary study. J Musculoskelet Pain. 2009;17(2):162-172. doi:10.1080/10582450902820572 4. Calvo-Lobo C, Pacheco-Da-Costa S, Hita-Herranz E. Efficacy of deep dry needling on latent myofascial trigger points in older adults with nonspecific shoulder pain: A randomized, controlled clinical trial pilot study. J Geriatr Phys Ther. 2017;40(2):63-73. doi:10.1519/JPT.0000000000000048 5. Calvo-Lobo C, Pacheco-Da-Costa S, Martnez-Martnez J, Rodrguez-Sanz D, Cuesta-lvaro P, Lpez-Lpez D. Dry needling on the infraspinatus latent and active myofascial trigger points in older adults with nonspecific shoulder pain: a randomized clinical trial. J Geriatr Phys Ther. 2018;41(1):1-13. doi:10.1519/JPT.0000000000000079 6. Campa-Moran I, Rey-Gudin E, Fernndez-Carnero J, et al. Comparison of dry needling versus orthopedic manual therapy in patients with myofascial chronic neck pain: A single-blind, randomized pilot study. Pain Res Treat. 2015;2015. doi:10.1155/2015/327307 7. Chou LW, Hsieh YL, Kao MJ, Hong CZ. Remote Influences of Acupuncture on the Pain Intensity and the Amplitude Changes of Endplate Noise in the Myofascial Trigger Point of the Upper Trapezius Muscle. Arch Phys Med Rehabil. 2009;90(6):905-912. doi:10.1016/j.apmr.2008.12.020 8. Chou L-W, Hsieh Y-L, Chen H-S, Hong C-Z, Kao M-J, Han T-I. Remote therapeutic effectiveness of acupuncture in treating myofascial trigger point of the upper trapezius muscle. Am J Phys Med Rehabil. 2011;90(12):1036-1049. doi:10.1097/PHM.0b013e3182328875 9. Chu J. Does EMG (dry needling) reduce myofascial pain symptoms due to cervical nerve root irritation? Electromyogr Clin Neurophysiol. 1997;37(5):259-272. 10. Cotchett MP, Munteanu SE, Landorf KB. Effectiveness of Trigger Point Dry Needling for Plantar Heel Pain: A Randomized Controlled Trial. Phys Ther. 2014;94(8):1083-1094. doi:10.2522/ptj.20130255 92 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 93 11. Couto C, De Souza ICC, Torres ILS, Fregni F, Caumo W. Paraspinal stimulation combined with trigger point needling and needle rotation for the treatment of myofascial pain: A randomized sham-controlled clinical trial. Clin J Pain. 2014;30(3):214-223. doi:10.1097/AJP.0b013e3182934b8d 12. Draolu D, Vural M, Karan A, et al. Effectiveness of dry needling for the treatment of temporomandibular myofascial pain: A double-blind, randomized, placebo controlled study. J Back Musculoskelet Rehabil. 2012;25(4):285-290. doi:10.3233/BMR-2012-0338 13. Dragoo JL, Wasterlain AS, Braun HJ, Nead KT. Platelet-rich plasma as a treatment for patellar tendinopathy: A double-blind, randomized controlled trial. Am J Sports Med. 2014;42(3):610-618. doi:10.1177/0363546513518416 14. Edwards J, Knowles N. Superficial Dry Needling and Active Stretching in the Treatment of Myofascial Pain - A Randomised Controlled Trial. Acupunct Med. 2003;21(3):80-86. doi:10.1136/aim.21.3.80 15. Eftekharsadat B, Babaei-Ghazani A, Zeinolabedinzadeh V. Dry needling in patients with chronic heel pain due to plantar fasciitis: A single-blinded randomized clinical trial. Med J Islam Repub Iran. 2016;30(401):1-9. 16. Erolu PK. A Comparison of the Efficacy of Dry Needling, Lidocaine Injection, and Oral Flurbiprofen Treatments in Patients with Myofascial Pain Syndrome: A Double-Blind (For Injection, Groups Only), Randomized Clinical Trial. Turkish J Rheumatol. 2013;28(1):38-46. doi:10.5606/tjr.2013.2779 17. Fernndez-Carnero J, La Touche R, Ortega-Santiago R, et al. Short-term effects of dry needling of active myofascial trigger points in the masseter muscle in patients with temporomandibular disorders. J Orofac Pain. 2010;24(1):106-112. 18. Ga H, Choi J-H, Park C-H, Yoon H-J. Dry Needling of trigger points with and without paraspinal needling in myofascial pain syndromes in elderly patients. J Altern Complement Med. 2007;13(6):617-624. doi:10.1089/acm.2006.6371 19. Ga H, Koh H-J, Choi J-H, Kim C-H. Intramuscular and nerve root stimulation vs lidocaine injection of trigger points in myofascial pain syndrome. J Rehabil Med. 2007;39:374-378. doi:10.2340/16501977-0058 20. Garvey TA, Marks MR, Wiesel SW. A prospective, randomized, double-blind evaluation of trigger-point injection therapy for low-back pain. Spine (Phila Pa 1976). 1989;14(9):962-964. doi:10.1097/00007632-198909000-00008 21. Gonzalez-Perez LM, Infante-Cossio P, Granados-Nuez M, Urresti-Lopez FJ. Treatment of temporomandibular myofascial pain with deep dry needling. Med Oral Patol Oral Cir Bucal. 2012;17(5):781-786. doi:10.4317/medoral.17822 93 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 94 22. Gonzalez-Perez LM, Infante-Cossio P, Granados-Nunez M, Urresti-Lopez FJ, Lopez-Martos R, Ruiz-Canela-Mendez P. Deep dry needling of trigger points located in the lateral pterygoid muscle: Efficacy and safety of treatment for management of myofascial pain and temporomandibular dysfunction. Med Oral Patol Oral Cir Bucal. 2015;20(3):e326-e333. doi:10.4317/medoral.20384 23. Hong C-Z. Lidocaine injection versus dry needling to myofascial trigger point. Am J Phys Med Rehabil. 1994;73(4):256-263. doi:10.1097/00002060-19940700000006 24. Hsieh Y-L, Kao M-J, Kuan T-S, Chen S-M, Chen J-T, Hong C-Z. Dry needling to a key myofascial trigger point may reduce the irritability of satellite MTrPs referred pain. Am J Phys Med Rehabil. 2007;86(5):397-403. doi:10.1097/PHM.0b013e31804a554d 25. Huang Y-T, Lin S-Y, Neoh C-A, Wang K-Y, Jean Y-H, Shi H-Y. Dry needling for myofascial pain: prognostic factors. J Altern Complement Med. 2011;17(8):755762. doi:10.1089/acm.2010.0374 26. Huguenin L, Brukner PD, McCrory P, Smith P, Wajswelner H, Bennell K. Effect of dry needling of gluteal muscles on straight leg raise: A randomised, placebo controlled, double blind trial. Br J Sports Med. 2005;39(2):84-90. doi:10.1136/bjsm.2003.009431 27. Ilbuldu E, Cakmak A, Disci R, Aydin R. Comparison of Laser, Dry Needling, and Placebo Laser Treatments in Myofascial Pain Syndrome. Photomed Laser Surg. 2004;22(4):306-311. doi:10.1089/pho.2004.22.306 28. Irnich D, Behrens N, Gleditsch JM, et al. Immediate effects of dry needling and acupuncture at distant points in chronic neck pain: Results of a randomized, double-blind, sham-controlled crossover trial. Pain. 2002;99(1-2):83-89. doi:10.1016/S0304-3959(02)00062-3 29. Itoh K, Katsumi Y, Hirota S, Kitakoji H. Effects of trigger point acupuncture on chronic low back pain in elderly patients - A sham-controlled randomised trial. Acupunct Med. 2006;24(1):5-12. doi:10.1136/aim.24.1.5 30. Itoh K, Katsumi Y, Hirota S, Kitakoji H. Randomised trial of trigger point acupuncture compared with other acupuncture for treatment of chronic neck pain. Complement Ther Med. 2007;15:172-179. doi:10.1016/j.ctim.2006.05.003 31. Itoh K, Katsumi Y, Kitakoji H. Trigger point acupuncture treatment of chronic low back pain in elderly patients - A blinded RCT. Acupunct Med. 2004;22(4):170-177. doi:10.1136/aim.22.4.170 32. Itoh K, Saito S, Sahara S, Naitoh Y, Imai K, Kitakoji H. Randomized trial of trigger point acupuncture treatment for chronic shoulder pain: A preliminary study. J Acupunct Meridian Stud. 2014;7(2):59-64. doi:10.1016/j.jams.2013.02.002 94 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 95 33. Kamanli A, Kaya A, Ardicoglu O, Ozgocmen S, Zengin FO, Bayik Y. Comparison of lidocaine injection, botulinum toxin injection, and dry needling to trigger points in myofascial pain syndrome. Rheumatol Int. 2005;25(8):604-611. doi:10.1007/s00296-004-0485-6 34. Kheradmandi A, Ebrahimian M, Ghaffarinejad F, Ehyaii V. The Effect of Dry Needling of the Trigger Points of Shoulder Muscles on Pain and Grip Strength in Patients with Lateral Epicondylitis : A Pilot Study. J Rehabil Sci Res. 2016;3(2015):58-62. 35. Liu W, Xue L, Tian M, Zhang F. Clinical research on a myofascial pain trigger point combining baihui acupoint therapy of myofascial pain syndrome and living quality analysis. Int J Clin Exp Med. 2016;9(2):866-876. 36. Llamas-Ramos R, Pecos-Martn D, Gallego-Izquierdo T, et al. Comparison of the short-term outcomes between trigger point dry needling and trigger point Manual therapy for the management of chronic mechanical neck pain: A randomized clinical trial. J Orthop Sport Phys Ther. 2014;44(11):852-861. doi:10.2519/jospt.2014.5229 37. Ma C, Wu S, Li G, Xiao X, Mai M, Yan T. Comparison of miniscalpel-needle release, acupuncture needling, and stretching exercise to trigger point in myofascial pain syndrome. Clin J Pain. 2010;26(3):251-257. doi:10.1097/AJP.0b013e3181b8cdc8 38. Macdonald AJR, Macrae KD, Master BR, Rubin AP. Superficial acupuncture in the relief of chronic low back pain. A placebo-controlled randomised trial. Ann R Coll Surg Engl. 1983;65(1):44-46. 39. Mahmoudzadeh A, Rezaeian Z, Karimi A, Dommerholt J. The effect of dry needling on the radiating pain in subjects with discogenic low-back pain: A randomized control trial. J Res Med Sci. 2016;21(94):1-7. doi:10.4103/17351995.192502 40. Mayoral O, Salvat I, Martn MT, et al. Efficacy of myofascial trigger point dry needling in the prevention of pain after total knee arthroplasty: A randomized, double-blinded, placebo-controlled trial. Evidence-Based Complement Altern Med. 2013;2013. 41. McMillan AS, Nolan A, Kelly PJ. The efficacy of dry needling and procaine in the treatment of myofascial pain in the jaw muscles. J Orofac Pain. 1997;11(4):307314. 42. Mejuto-Vzquez MJ, Salom-Moreno J, Ortega-Santiago R, Truyols-Domnguez S, Fernndez-de-las-Peas C. Short-term changes in neck pain, widespread pressure pain sensitivity, and cervical range of motion after the application of trigger point dry needling in patients with acute mechanical neck pain: A randomized clinical trial. J Orthop Sport Phys Ther. 2014;44(4):252-260. doi:10.2519/jospt.2014.5108 95 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 96 43. Myburgh C, Hartvigsen J, Aagaard P, Holsgaard-larsen A. Skeletal muscle contractility , self-reported pain and tissue sensitivity in females with neck / shoulder pain and upper Trapezius myofascial trigger points a randomized intervention study. Chiropr Man Ther. 2012;20(36):1-10. doi:10.1186/2045-709X20-36 44. Pecos-Martn D, Montaez-Aguilera FJ, Gallego-Izquierdo T, et al. Effectiveness of dry needling on the lower trapezius in patients with mechanical neck pain: A randomized controlled trial. Arch Phys Med Rehabil. 2015;96(5):775-781. 45. Prez-Palomares S, Olivn-Blzquez B, Magalln-Botaya R, et al. Percutaneous electrical nerve stimulation versus dry needling: Effectiveness in the treatment of chronic low back pain. J Musculoskelet Pain. 2010;18(1):23-30. doi:10.3109/10582450903496047 46. Rayegani SM, Bayat M, Bahrami MH, Raeissadat SA, Kargozar E. Comparison of dry needling and physiotherapy in treatment of myofascial pain syndrome. Clin Rheumatol. 2014;33(6):859-864. doi:10.1007/s10067-013-2448-3 47. Rha D, Park G-Y, Kim Y-K, Kim MT, Lee SC. Comparison of the therapeutic effects of ultrasound-guided platelet-rich plasma injection and dry needling in rotator cuff disease: a randomized controlled trial. Clin Rehabil. 2013;27(2):113122. doi:10.1177/0269215512448388 48. Salom-Moreno J, Jimnez-Gmez L, Gmez-Ahufinger V, et al. Effects of lowload exercise on post-needling induced pain after dry needling of active trigger point in individuals with subacromial pain syndrome. PM&R. 2017;9(12):12081216. doi:10.1016/j.pmrj.2017.04.012 49. Santos RB da C, Carneiro MIS, Oliveira DM de, Maciel AB do R, Monte-Silva KK do, Arajo M das GR. Impact of dry needling and ischemic pressure in the myofascial syndrome: controlled clinical trial. Fisioter em Mov. 2014;27(4):515522. doi:10.1590/0103-5150.027.004.AO03 50. Silva ROF da, Conti PCR, Araujo C dos RP, Silva R dos S. Evaluation of dry needling and 0.5% lidocaine injection therapies in myofascial pain trigger points in masticatory muscles. Dental Press J Orthod. 2012;17(2):113-118. 51. Srbely JZ, Dickey JP, Lee D, Lowerison M. Dry needle stimulation of myofascial trigger points evokes segmental anti-nociceptive effects. J Rehabil Med. 2010;42:463-468. doi:10.2340/16501977-0535 52. Sterling M, Vicenzino B, Souvlis T, Connelly LB. Dry-needling and exercise for chronic whiplash-associated disorders: a randomized single-blind placebocontrolled trial. Pain. 2015;156(4):635-643. doi:10.1097/01.j.pain.0000460359.40116.c1 96 CONFLICTS OF INTEREST AND RESEARCHER ALLEGIANCE IN DRY NEEDLING TRIALS 97 53. Tekin L, Akarsu S, Durmu O, akar E, Diner U, Kiralp MZ. The effect of dry needling in the treatment of myofascial pain syndrome: A randomized doubleblinded placebo-controlled trial. Clin Rheumatol. 2013;32(3):309-315. http://dx.doi.org/10.1007/s10067-012-2112-3. 54. Tllez-Garca M, De-la-Llave-Rincn AI, Salom-Moreno J, Palacios-Cea M, Ortega-Santiago R, Fernndez-de-Las-Peas C. Neuroscience education in addition to trigger point dry needling for the management of patients with mechanical chronic low back pain: A preliminary clinical trial. J Bodyw Mov Ther. 2015;19(3):464-472. doi:10.1016/j.jbmt.2014.11.012 55. Tough EA, White AR, Richards SH, Campbell JL. Myofascial trigger point needling for whiplash associated pain - A feasibility study. Man Ther. 2010;15(6):529-535. doi:10.1016/j.math.2010.05.010 56. Tsai C-T, Hsieh L-F, Kuan T-S, Kao M-J, Chou L-W, Hong C-Z. Remote Effects of Dry Needling on the Irritability of the Myofascial Trigger Point in the Upper Trapezius Muscle. Am J Phys Med Rehabil. 2010;89(2):133-140. doi:10.1097/PHM.0b013e3181a5b1bc 57. Uemoto L, Garcia M, Gouvea C, Vilella O, Almeida Alfaya T. Laser therapy and needling in myofascial trigger point deactiviation. J Oral Sci. 2013;55(2):175-181. 58. Venncio R de A, Alencar Jr FGP, Zamperini C. Different Substances and Dry Needling Injections in Patients with Myofascial Pain and Headaches. J Craniomandib Sleep Pract. 2008;26(2):96-103. doi:10.1179/crn.2008.014 59. Venancio R de A, Alencar Jr. FGP, Zamperini C. Botulinum toxin, lidocaine, and dry-needling injections in patients with myofascial pain and headaches. J Craniomandib Pract. 2009;27(1):1-8. 60. Yeganeh Lari A, Okhovatian F, Naimi S sadat, Baghban AA. The effect of the combination of dry needling and MET on latent trigger point upper trapezius in females. Man Ther. 2016;21:204-209. doi:10.1016/j.math.2015.08.004 61. Zheng Y, Shi D, Wu X, et al. Ultrasound-guided miniscalpel-needle release versus dry needling for chronic neck pain: a randomized controlled trial. Evidence-Based Complement Altern Med. 2014;235817:235817. 62. Ziaeifar M, Arab AM, Karimi N, Nourbakhsh MR. The effect of dry needling on pain, pressure pain threshold and disability in patients with a myofascial trigger point in the upper trapezius muscle. J Bodyw Mov Ther. 2014;18(2):298-305. doi:10.1016/j.jbmt.2013.11.004 97  ...

... Comparing Traditional Modeling Approaches Versus Predictive Analytics Methods for Predicting Multiple Sclerosis Relapse and All-Cause Urgent Care Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Karen Walsh, MS, MBA Copyright December 2, 2020 By: Karen Walsh, MS, MBA All rights reserved Approved by: Elizabeth Moore, PhD Committee Chair ______________________________ Brant J. Oliver, PhD, MS, MPH, FNP-BC, PMHN-BC Committee Member ______________________________ Johnathon Kyle Armstrong, PhD Committee Member ______________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program Chair, Interprofessional Health & Aging Studies University of Indianapolis ______________________________ Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis ______________________________ Running head: STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Comparing Traditional Modeling Approaches versus Predictive Analytics Methods for Predicting Multiple Sclerosis Relapse and All-Cause Urgent Care Karen Walsh University of Indianapolis 1 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 2 Abstract Multiple sclerosis is a complex and costly chronic (3C) condition that currently has no cure. In a condition like multiple sclerosis, which has an unpredictable course, the use of predictive analytics could help health systems learn better, faster, and to improve more effectively and predict rather than react to emerging health needs for people with MS. This study compared traditional statistical methods to different predictive analytics methods on two separate endpoints, MS relapse and all-cause urgent care. Binary logistic regression was compared with other machine learning models, specifically ridge, least absolute shrinkage and selection operator (LASSO), and random forest. Results indicated when comparing relapse indices across models random forest significantly outperformed logistic regression and other machine learning algorithms (perfA =27.1%, perfM =27.5%). However, for perfF,, logistic regression and random forest performed relatively the same. Ridge and LASSO outperformed logistic regression (perfM1 =0.9%, perfM2 =9.4%, perfF2=25.8%) respectively. Results indicated when comparing all-cause urgent care indices across models, logistic regression performed similarly to random forest and LASSO (perfA = -1.1%, perfM =1.58%, perfA1 = -0.5%). Ridge performed worse overall compared to logistic regression (perfA2 = -17.8%, perfM2 = -3.84%). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 3 Acknowledgements My doctoral journey is non-traditional, and began in 2014 at Thomas Jefferson University as a PhD student and wrapped up seven years later at University of Indianapolis. This long journey was due to family priorities and a big corporate job which made it almost impossible to move forward as a doctoral student. Thomas Jefferson University, College of Population Health gave me a job opportunity as a faculty member in 2018, to continue to move forward and complete my doctorate. First and foremost, I am extraordinary grateful to my husband, Thomas Walsh, my biggest supporter and fan throughout this time. Not only did he support me during my doctoral journey, but a second masters degree. His willingness to help with family and household obligations, when I was too busy studying, writing, or doing homework was invaluable. He was also a shoulder to cry on when I was overwhelmed by all of my responsibilities, and always found a way to make me laugh. You are the best. I would also like to thank my chair, Dr. Moore, who has been supportive during this process and has made me appreciate APA formatting and become a better academic writer. I want to also thank Dr. Oliver, not only as a committee member, but as a mentor. I started working with Dr. Oliver in 2018 with MS-CQI and he has been instrumental on guiding and supporting me in my academic growth and an extraordinary research experience. My final acknowledgement goes to Dr. Kyle Armstrong, who pushed me to the next methodological level from a data science perspective during this journey. I am grateful for the time he spent with me to better understand complex data science paradigms and R programming language. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 4 Table of Contents List of Tables .................................................................................................................................. 6 List of Figures ................................................................................................................................. 8 Chapter 1 Introduction .................................................................................................................. 10 Clinical Quality Improvement .......................................................................................... 11 Predictive Analytics .......................................................................................................... 12 Problem Statement ............................................................................................................ 13 Purpose Statement............................................................................................................. 14 Literature Review.............................................................................................................. 14 Chapter 2 Method ......................................................................................................................... 29 Parent Study Design .......................................................................................................... 29 Proposed Study Design ..................................................................................................... 30 Participants........................................................................................................................ 30 Setting ............................................................................................................................... 31 Data Management ............................................................................................................. 31 Procedures ......................................................................................................................... 31 Continuous Quality Improvement Intervention ................................................................ 32 Data and Data Collection for Relapses and All-Cause Hospitalization............................ 32 Exploratory Data Analysis ................................................................................................ 33 Predictive Models ............................................................................................................. 34 Chapter 3 Results .......................................................................................................................... 42 Center and Participant Characteristics .............................................................................. 42 Clinical Outcomes and Center Level Variation ................................................................ 42 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 5 Training and Testing Datasets .......................................................................................... 43 Quantitative Aim 1: Relapse Predictive Model Comparison ............................................ 43 Quantitative Aim 2: All-Cause Utilization Predictive Model Comparison ...................... 49 Chapter 4 Discussion and Conclusions ......................................................................................... 54 Quantitative Discussion Aim 1: Relapse Predictive Model Comparison ......................... 56 Quantitative Discussion Aim 2: All-Cause UC Predictive Model Comparison ............... 58 Limitations ........................................................................................................................ 59 Implications on Future Research ...................................................................................... 61 Conclusion ........................................................................................................................ 62 References ......................................................................................Error! Bookmark not defined. Appendix A IRB Approval ......................................................................................................... 118 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 6 List of Tables Table 1. Patient Characteristics for Year 3 by Center .................................................................. 73 Table 2. Outcomes for Year 3 by Center ...................................................................................... 74 Table 3. Patient Characteristics and Outcomes by Derivation and Validation Cohort................. 75 Table 4. Bivariate Analysis (n = 2,532) ........................................................................................ 78 Table 5. Model Comparison Metrics ............................................................................................ 79 Table 6. Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Stepwise Logistic Regression ......................................................................................................................... 80 Table 7. Relapse Model Comparison Metrics-Cross-Validated (Accuracy) LASSO................... 81 Table 8. Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Ridge ...................... 82 Table 9. Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Random Forest ....... 83 Table 10. Relapse Model Comparison Metrics-Cross-Validated (MCC) LASSO ....................... 84 Table 11. Relapse Model Comparison Metrics-Cross-Validated (MCC) Ridge .......................... 85 Table 12. Relapse Model Comparison Metrics-Cross-Validated (MCC) Random Forest ........... 86 Table 13. Relapse Model Comparison Metrics-Cross-Validated (F1) LASSO ............................ 87 Table 14. Relapse Model Comparison Metrics-Cross-Validated ................................................. 88 Table 15. Relapse Model Comparison Metrics-Cross-Validated (F1) Random Forest ................ 89 Table 16. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Stepwise Logistic Regression ........................................................................................... 90 Table 17. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) LASSO .............................................................................................................................. 91 Table 18. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Ridge ................................................................................................................................. 92 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 7 Table 19. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Random Forest .................................................................................................................. 93 Table 20. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (MCC) Ridge .. 94 Table 21. All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (MCC) Random Forest................................................................................................................................. 95 Table 22. Relapse Model Comparison Metrics-All Predictive Models ........................................ 96 Table 23. All-Cause Urgent Care Model Comparison Metrics-All Predictive Models................ 97 Table 24. Performance Relapse Indices ........................................................................................ 98 Table 25. Performance All-Cause Urgent Care Indices................................................................ 99 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 8 List of Figures Figure 1. A Diagram of IHI Model for Improvement Showing Plan-Do-Study-Act process. ... 100 Figure 2. Representation of the Tradeoff Between Flexibility and Interpretability Using Different Statistical Learning Methods. ......................................................................................... 101 Figure 3. Ridge Regression Coefficient Estimates for Each Value of . .................................... 102 Figure 4. Box Plot of Age by Center. ......................................................................................... 103 Figure 5. Relapse ROC Curve Model Comparison Optimized for Accuracy. ............................ 104 Figure 6. Relapse Gain Curve Model Comparison Optimized for Accuracy. ............................ 105 Figure 7. Relapse ROC Curve Model Comparison Optimized for MCC. ................................. 106 Figure 8. Relapse Gain Curve Model Comparison Optimized for MCC. .................................. 107 Figure 9. Relapse ROC Curve Model Comparison Optimized for F1. ....................................... 108 Figure 10. Relapse Gain Curve Model Comparison Optimized for F1. ..................................... 109 Figure 11. All-Cause Urgent Care ROC Curve Model Comparison Optimized for Accuracy. . 110 Figure 12. All-Cause Urgent Care Gain Curve Model Comparison Optimized for Accuracy. .. 111 Figure 13. All-Cause Urgent Care ROC Curve Model Comparison Optimized for MCC. ........ 112 Figure 14. All-Cause Urgent Care ROC Curve Model Comparison optimized for MCC. ......... 113 Figure 15. Relapse Model Measures Comparison Including All Optimizations (Accuracy, MCC, F1). .................................................................................................................................. 114 Figure 16. All Relapse Model ROC Curves Including All Optimizations (Accuracy, MCC, F1). .................................................................................................................................. 115 Figure 17. All-Cause Urgent Care Model Measures Comparison Including All Optimizations (Accuracy, MCC). ........................................................................................................... 116 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 9 Figure 18. All-Cause Urgent Care Models ROC Curves Including All Optimizations (Accuracy, MCC). ............................................................................................................................. 117 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 10 Chapter 1 Introduction Multiple sclerosis (MS) is one of the most common chronic neurological conditions for adults, with a prevalence of nearly one million people in the United States (Wallin et al., 2019). While the exact pathologic mechanism for MS remains incompletely understood, it involves an immune-mediated process in which an abnormal response of the bodys immune system is directed against the central nervous system (CNS; National Multiple Sclerosis Society, 2019a). MS is a complex and costly chronic (3C) condition that currently has no cure. The disease course is often debilitating, disabling, and unpredictable and causes functional and symptomatic impairments including sensory, motor, cognitive, and psychiatric problems and debilitating fatigue (Bozkaya, Livingston, Migliaccio-Walle, & Odom, 2017). There are four different types of MS, relapsing-remitting (RRMS), primary progressive (PPMS), secondary progressive (SPMS), and progressive relapsing (PRMS; National Multiple Sclerosis Society, 2019b). Relapsing-remitting MS is the most common disease course with 85% of people initially diagnosed with RRMS (National Multiple Sclerosis Society, 2019b). This type of MS is characterized by relapses of new or increasing neurologic symptoms which are followed by periods of complete or partial recovery (National Multiple Sclerosis Society, 2019b). Primary progressive MS is defined as worsening neurologic function from the onset of symptoms without early relapses or remissions. About 15% of people with MS have PPMS (National Multiple Sclerosis Society, 2019b). Secondary progressive MS follows an initial relapsing-remitting disease course. People that have RRMS may at some point transition to SPMS where there is a progressive worsening of neurologic function over time (National Multiple Sclerosis Society, 2019b). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 11 MS causes a socioeconomic burden to the individual and society due to the disability of the disease (O'Connell et al., 2014). The average total cost of care over 12 months for individuals with MS was $51,692 for non-relapsing MS, while the cost for relapse-remitting MS was $58,648 (Jones, Pike, Marshall, & Ye, 2016) The most common form of this disease is relapsing MS with a relapse of approximately one per year (Vollmer, 2007). Research shows that participants with relapsing MS had significantly higher annual costs for physician consultations ($464; p < .05), hospitalizations ($3,693; p < .05) and total MS costs ($4,390; p < .05) when compared to participants with non-relapsing MS (Iezzoni & Ngo, 2007; Jones et al., 2016). Clinical Quality Improvement The proposed study is part of an MS clinical quality improvement research collaborative, the Multiple Sclerosis Continuous Quality Improvement (MS-CQI) Collaborative, which is the parent study (See Appendix A). MS-CQI is an improvement research collaborative that employs a combination of an improvement collaborative structure and a step-wedge randomized design to test the comparative effectiveness of quality improvement (QI) intervention versus usual care. The Health Foundation (2009) describes a QI collaborative as requiring five critical features: (1) there is a specified topic; (2) clinical experts and experts in quality improvement provide ideas and support for improvement; (3) there are multi-professional teams from multiple sites; (4) there is a model for improvement that focuses on setting clear and measurable targets; (5) the collaborative process involves a series of structured activities (de Silva, 2014; Hulscher, Schouten, & Grol, 2009). Clinical QI collaboratives can facilitate systems-level change in real-world settings (de Silva, 2014; Hulscher et al., 2009.), but data from these collaboratives need to be systematically vetted and analyzed using rigorous research designs. MS-CQI is implementing descriptive STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 12 statistics as well as inferential outcomes analysis across four centers to compare performance. Statistical process control (SPC) benchmarking with variation analysis by center is also being conducted. Predictive analytics and more advanced statistical techniques can complement and may improve upon this current standard. Predictive Analytics The proposed research study expanded upon the learning health systems (LHS) model even further by moving this work towards predictive analytics. MS-CQI was created as the first LHS improvement research collaborative for MS, and is addressing some of these deficiencies by utilizing a learning health system model that employs integrated improvement and research methods, including individual and population levels of analysis and a randomized comparative effectiveness design (Oliver, 2019). MS-CQI represents the vanguard of a shift towards the inclusion of systems-level approaches to improve MS care and demonstrate value. The Institute of Medicine (IOM) and the Institute for Healthcare Improvement (IHI) have called for a new systems-oriented focus and continuous improvement culture in the United States (IOM, 2001; IHI, n.d.). Nelsons Balanced Measures Clinical Value Compass framework is commonly used in healthcare QI because it can specify process, measure systems-level quality, and value (Lindblad et al., 2017; Nelson et al., 1995). The framework has four categories: (a) clinical outcomes; (b) functional health; (c) patient experience and satisfaction; and (d) utilization (Nelson et al., 2016). Predictive analytics could optimize the feed-forward and feedback aspects of the MS LHS. In a condition like MS, which has an unpredictable course, the use of predictive analytics could help health systems learn better, faster, and to improve more effectively and predict rather than react to emerging health needs for people with MS. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 13 Predictive analytics introduces a new culture in data analysis. The evolution of algorithmic modeling has grown tremendously outside of the field of statistics (Breiman, 2001). These algorithms can be used on various datasets and be more accurate than existing data modeling approaches (Breiman, 2001). Model selection is different in predictive analytics versus traditional statistical modeling approaches. Traditional statistical methods allow one to determine, based on subject matter knowledge and/or simple descriptive or inferential statistics, what predictors to include in a model. Some of these methods include linear regression and logistic regression (James, Witten, Hastie, & Tibshirani, 2013). In predictive analytics the combination of test/training data splits or cross-validation, one can assess model error reliably (James et al., 2013). Predictive analytics methods, including regularization, and ensemble approaches can deal with far larger numbers of predictors in an automated fashion (James et al., 2013). Some of these predictive analytics methods include ridge regression and least absolute shrinkage and selection operator (LASSO) regression (James et al., 2013). Ridge regression and LASSO are both shrinkage methods and can be used with many features in the model without overfitting (James et al., 2013). Overfitting is when the model is trained and fits well on the training dataset, but performs poorly on the test dataset and becomes less generalizable. Both of these models shrink or regularize the coefficient estimates or shrinks them towards zero (James et al., 2013). By shrinking coefficient estimates, this improves the fit of the model by reducing the models variance (James et al., 2013). Problem Statement Historically, MS care has been studied and improved through basic and clinical trials research and epidemiological studies on population outcomes and quality of life. MS registries such as NARCOMS, MS PATH, and MS link have been developed. Only very recently has MS STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 14 entered the realm of improvement science and population health. The MS-CQI is the first multicenter exemplar using a learning health system model that employs integrated improvement and research methods, including individual and population levels of analysis (Oliver, 2019). In comparison, the MS-Advance study was a single center, non-randomized study and used a patient centered medical home as their intervention (Meninno et al., 2018). A multi-center approach is needed for a successful collaborative model. There is a dearth of research on predicting relapse in MS QI collaboratives using predictive analytics, as well as predicting MS all-cause UC utilization. This study is significant because it analyzes outcomes of a multicenter MS quality improvement collaborative as well as comparing traditional statistical methods to predictive analytics for predicting MS relapse and all-cause urgent care (UC) utilization. Relapse and UC utilization are aspects of MS that are costly and disabling, and to date, have been largely unpredictable. For this study all-cause, UC utilization included all-cause emergency department (ED) visits and all-cause UC visits. Purpose Statement This study focused on predictive modeling of MS relapse and all-cause UC utilization by comparing standard statistical and predictive analytics approaches. The first research aim is to determine whether results are similar between traditional statistical methods and predictive analytics methods when predicting MS relapse. The second research aim is to determine whether traditional statistical methods versus predictive analytics methods results are similar when predicting MS all cause-UC utilization. Literature Review This study analyzes outcomes of a multicenter MS quality improvement collaborative by comparing traditional statistical methods to predictive analytics for predicting MS relapse rate STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 15 and all-cause UC. The literature review summarizes available evidence on: (a) quality improvement collaboratives; (b) MS predictive modeling on relapse and all-cause UC utilization; and (c) studies on predictive model comparisons. Relapsing-remitting MS. Relapse-remitting MS accounts for approximately 85% of MS cases (Vollmer, 2017). RRMS is characterized by periods of exacerbations followed by partial or full recovery to prior baseline. Longitudinally, a subset of people with RRMS progress into a progressive stage called SPMS in which relapses cease, neurological progression continues, and recovery is rare (Vollmer, 2007). MS relapses occur on average about 1.1 per year early in the course of the disease; however, this rate appears to decrease as the disease progresses (Vollmer, 2007). Immunotherapy disease-modifying treatments (DMTs) can dramatically reduce the relapse rate as well as related brain magnetic resonance imaging (MRI) changes and functional neurological progression of disability (Stoppe Busch, Krizek, & Then Bergh, 2017); however, DMTs do not cure MS (Stoppe et al., 2017). Disease-modifying therapies are not used to control MS symptoms. In the early stages of MS, relapses are more likely to resolve to prior baseline. However, as the disease progresses, relapses result in incomplete recovery and progressive disability and can have a significant impact on the individuals quality of life (O'Connell et al., 2014). Relapses can result in economic cost of health services such as ED visits and hospitalizations (Iezzoni & Ngo, 2007). In addition, OConnell et al. (2014) found the largest component of indirect costs for an MS relapse was loss of income due to employment status. Relapses function independently of other clinical MS symptoms; however, relapses can worsen MS symptoms and accelerate disease progression (Koch-Henriksen, Thygesen, Srensen, & Magyari, 2019). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 16 Improvement science. MS has only recently entered the realm of improvement science and population health. Groundwork for this has been established by large epidemiological registries, such as the North American Research Committee on Multiple Sclerosis (NARCOMS), the Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATH), and Multiple Sclerosis Leadership and Innovation Network (MS LINK). NARCOMS is a data registry that utilizes patient experience to improve clinical care and quality of life (NARCOMS, 2017). MS PATH uses technology and collects MS patient data from routine office visits to enable MS research (MS PATH, 2016). MS LINK is an interdisciplinary research community that is focused on improving the care of an MS individual (Gisler, 2019). Finally, the Slifka Longitudinal MS Study researches population demographic and clinical measures of utilization, illness, and cost (Minden et al., 2006). Historically, MS care has been studied and improved at the basic and clinical trials research and epidemiological studies on outcomes and quality of life. These approaches have not studied MS care or outcomes at a system (MS Center/MS clinic) level. MS-CQI represents the vanguard of this approach utilizing an improvement science approach. The improvement science approach combines QI with rigorous scientific methodology. Continuous quality improvement. The QI approach is a system level approach influenced by the IHI, the IOM and the Affordable Care Act. The IHI triple aim specifically suggests a new systems-oriented focus and continuous improvement culture (IHI, n.d.). Also, the IOM shows significant quality and safety deficiencies in our current health system (Nationalacademies.org, 2019). To Err is Human: Building a Safer Health System (IOM, 2001) was published in 1999. The authors stated that tens of thousands of people in the United States die every year due to STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 17 mistakes that are preventable with their care (IOM, 2001). This publication led to a call for a more systems-level change and Crossing the Quality Chasm (IOM, 2001) was published with a focus on how the U.S. health system needs to be overhauled to improve the delivery of care to PwMS (Fanjiang, Grossman, Compton, & Reid, 2005). The IOM suggests six aims of improvement: safe, effective, patient-centered, timely, efficient and equitable (IOM, 2001). Six redesign imperatives were notated to drive these areas of improvement: (a) reengineered care processes; (b) effective use of information technologies; (c) knowledge and skills management; (d) development of effective teams; (e) coordination of care across patient conditions; (f) use of performance and outcomes measurement (IOM, 2001). The LHS model was popularized by this work, proposing that improvement and research could be simultaneously conducted and better outcomes achieved by focusing on system-level performance (Nelson et al., 1995). MS-CQI was created as the first LHS improvement research collaborative for MS, and is addressing some of these deficiencies by utilizing a learning health system model that employs integrated improvement and research methods. There are many popular QI approaches that are used nationally among CQI collaboratives such as the IHI Model for Improvement, Clinical Microsystems, and Lean Six Sigma (IHI, n.d.; Clinical Microsystems, n.d.; Lean Six Sigma Online, n.d.). These different methodologies are all utilized in healthcare settings across the United States and when implemented, can have significant results. The Clinical Microsystems approach focuses on microsystems, which are defined as the essential building blocks of larger health system (Clinical Microsystems, n.d.). Clinical microsystems complete value-added and hands-on work inside health organizations and are living units that have the patient and their needs, front and center (Clinical Microsystems, n.d.). This approach builds upon foundational work by business scholar P. Brian Quinn which STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 18 documented the competitive advantage of businesses that excelled in front-line service performance focused on optimizing the capability of smallest replicable units or SRUs improvement of SRU performance translated into improved overall performance (Quinn, 1992). The SRU theory was later adapted for healthcare by Batalden and Nelson to develop the microsystems approach. This approach guided the systems-level focus for the MS-CQI study. (Nelson, Batalden, Godfrey, & Lazar, 2011). Lean Six Sigmas primary goal is to reduce waste and variation to create an environment of optimal quality control (Lean Six Sigma Online, n.d.). This approach includes the use of SPC methods to study and improve performance variation these include the use of SPC for benchmarking in improvement initiatives and collaboratives (Benneyan, Lloyd, & Plsek, 2003). This approach influenced the benchmarking approach used in the MS-CQI study. The Model for Improvement is used by healthcare organizations as a framework to accelerate healthcare improvement through iterative small tests of change aimed at optimizing but not replacing the existing model (Institute for Healthcare Improvement: How to Improve, n.d.). The fundamental structure of the IHI Model for Improvement includes piloting the changes on a small-scale utilizing Plan-Do-Study-Act (PDSA) cycles, see Figure 1 (Institute for Healthcare Improvement: How to Improve, n.d.). The IHI model is influenced by Kolbs experiential learning model for adult learning and in turn, has influenced modern design approaches including human centered design and agile prototyping development (Kolb, 1984). This approach influenced the design of the improvement intervention for the MS-CQI study, which includes professional improvement coaching and a modified version of the IHI Breakthrough Series (BTS) collaborative structure (IHI, 2003) to guide microsystem teams applying PDSA cycles to improve system performance. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 19 System-level change focus. The current state of the healthcare system in the United States has recently moved towards a value-based quality of care (Porter, 2009). In particular, the Affordable Care Act is driving a shift from productivity to systems-level value-based reimbursement (healthcare.gov, 2019) and suggests that a new paradigm of continuous quality improvement will be required to optimize value and quality. It will be necessary to demonstrate the value of MS care at systems and population levels, especially in complex, high cost, chronic 3C conditions such as MS. De Silva (2014) reported that quality improvement collaboratives facilitate systems-level change in real-world settings. Regional and national CQI collaboratives that are utilizing these improvement methodologies have demonstrated significant results. The Northern New England Cardiovascular Network (NNE) Disease Study Group has used a shared data registry and QI methods to reduce mortality and morbidity across cardiac surgery centers in the U.S. (Northern New England Cardiovascular Network Disease Study Group, n.d.). Research on quality collaboratives for hypertension has also been successful. Participating Medicaid managed care plans in California reduced the hypertension of their patients (Backman et al., 2017). There was an increase of 5% in their rates of controlled hypertension (Backman et al., 2017). The Vermont Oxford Neonatal Health Network (VONHN) is a nonprofit collaboration of health care professionals as an interdisciplinary community to change the landscape neonatal care (Vermont Oxford Network, n.d.). The goal of VONHN is to improve the safety, quality, and value of care for newborn babies and their families through a collaboration program of education, quality improvement, and research (Vermont Oxford Network, n.d.). The Inflammatory Bowel Disease (IBD) Qorus is a collaboration between patients and healthcare STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 20 providers to improve quality care and improved outcomes for people with IBD (Crohns & Colitis Foundation, n.d.). The Cystic Fibrosis Foundation (CFF) Learning and Leadership Collaboratives (LLCs), where 110 centers have participated in a national QI collaborative for over a decade. The CFF utilizes QI methods as well as a systems-level registry. The CFF collaboratives have had success in reduced mortality, improved life expectancy, decreased morbidity, and developed several process quality indicators (Godfrey& Oliver, 2014; Marshall & Nelson, 2014; Mogayze et al., 2014; Sabadosa & Batalden, 2014). The differentiating characteristic of MS-CQI is that it includes a randomized research design. The prior collaboratives are cohort designs such as VONHN or do not have research designs (CFF, IBD). The MS-CQI is the vanguard effort for improvement science in MS because it builds upon these prior exemplars by merging QI and randomized research approaches. Additionally, current evidence suggests that improvement collaboratives and improvement coaching are equally effective in achieving success in clinical outcomes (Gustafson et al., 2013). Also, Gustafson and colleagues (2013), discuss that combining both coaching and improvement collaboratives optimize results since the components are additive. The MS-CQI intervention is based on the utilization of an improvement coach in combination with an improvement collaborative model. Parent study design. The parent study (MS-CQI) is utilizing a prospective step-wedge randomized design, which is also referred to as a Comprehensive Dynamic Trial (West et al., 2008). The step-wedge cluster randomized trial design has become increasingly popular within clinical trials, and can be used in the evaluation of service delivery type interventions (Hemming, Haines, Chilton, Girling, & Lilford, 2015). In addition, the step-wedge design allows for STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 21 comparisons within microsystems and can better accommodate smaller sample sizes (West et al., 2008). In a standard step-wedge design, clusters randomly and sequentially crossover from control to intervention until most or all of the clusters are exposed to the intervention. At the beginning of the trial, no clusters are exposed (Hemming et al., 2015). After a baseline period, cluster(s) are successively randomized in a step-wise fashion to go from control to intervention until most or all the clusters have been exposed (Hemming et al., 2015). Therefore, each cluster is part of the control and intervention, which allows robust scientific evaluation in shorter time periods and with fewer clusters than standard cluster-randomized designs (Hemming et al., 2015). The intervention employed by MS-CQI is a modified IHI Breakthrough Series clinical quality improvement (CQI) intervention, which augments the traditional BTS model by adding professional improvement coaching to accelerate front-line improvement capability (IHI, 2003; Oliver, Messier, & Hall, 2019). The IHI Breakthrough Series, CQI intervention with Professional Improvement Coaching was coordinated by clinical teams at participating sites under the guidance of a professional improvement coach utilizing the IHI improvement collaborative model. This intervention is a hybrid adaptation of the LLC and CFF (Godfrey & Oliver, 2014). The general structure and process, as outlined by and the IHI Breakthrough Series improvement collaborative model, was utilized for this intervention (Kilo, 1998; see Appendix A). The core MS-CQI study used descriptive statistics as well as inferential outcomes analysis to assess performance across the four centers and between exposure (QI) and control (usual practice) conditions. The SPC benchmarking with variation analysis by center is also being conducted. The proposed study built upon the parent study and focused on predictive STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 22 analytics and more advanced statistical techniques that can complement and improve upon current MS-CQI analysis. Predictive statistical modeling. In MS, relapses have been a focal point of research; however, there is a dearth of research on predicting MS relapses. Research has been used to determine predictors of MS relapse rate utilizing Poisson regression using demographic, clinical, and MRI variables (Held, Heigenhauser, Shang, Kappos, & Polman, 2005). Held et al.s (2005) study included 821 participants from the placebo group of the Sylvia Lawry Centre for Multiple Sclerosis Research (SLCMSR) and was aimed at determining prognostic factors available at baseline to the on-study relapse rate in MS. The timeframe of the study included 24 months prior to the study baseline and the MS disease course. The univariate analysis included 10 predictors for predicting the on-study relapse rate (Held et al., 2005). Results of the univariate analysis show that the on-study relapse rate was higher for younger female participants, for relapseremitting participants versus secondary progressive participants, and for participants with positive enhancement status at entry using Wilcoxon test, p < .05 (Held et al., 2005). A higher relapse rate was also associated with a shorter disease duration, lower entry EDSS, more prestudy relapses, and more enhancing lesions in T1 (Held et al., 2005). A Poisson regression model was also fitted using five predictors to predict the on-study relapse rate (Held et al., 2005). The fitted Poisson model results showed that disease duration (estimate = -0.02) and previous relapse number were significant (p < .05) predictors (estimate = 0.59 for one, 0.91 for two, and 1.45 for three or more relapses vs no relapses; Held et al., 2005). While the Poisson regression is a solid traditional model for a continuous outcome, other sophisticated predictive modeling approaches could have been evaluated. In addition, only including 10 features in the model can be limiting in increasing accuracy and optimization. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 23 Research has also been done on predicting RRMS utilizing discrete distribution models to characterize relapsing-remitting contrast-enhancing lesions (CEL) to quantify the interpatient variability (Velez de Mendizabal et al., 2013). Mendizabal et al.s (2013) study analyzed nine MS participants over 48 months that had a monthly MRI. Sixteen structural models that included seven different probability distributions were evaluated with a maximum of six predictors (Velez de Mendizabal et al., 2013). Results showed that based on the number of model parameters and the precision of the parameters the best fitting model was the negative binomial compared to the other models such as Poisson model, Poisson model with mixed distribution, Zero-Inflated and Generalized Poisson models and the Zero-Inflated Negative Binomial model (Velez de Mendizabal et al., 2013). The minimum value of the objective function, which is approximate to -2 log (likelihood) [-2LL], was criteria used as model comparison during model development (Velez de Mendizabal et al., 2013). The Akaike information criteria (AIC) was also used as measurement in the final model. In addition, the final model was based on the precision of parameter estimates and the results from model predictive performance from model simulations were compared (Velez de Mendizabal et al., 2013). While Velez de Mendizabal and colleagues model compared 16 different structural models, the sample size was extremely small (n = 9) and results may not be generalizable to the MS population. Measurement on evaluating the best performing model used was a strength of the study, however, traditional statistical models were used and machine algorithms were excluded, which leaves a gap in the model evaluation of the study. There is also some evidence on implementing a score to identify individual PwMS with RRMS via a multivariate Cox analysis, where the first relapse was the independent variable to try and predict future relapses in the short term (Sormani, Rovaris, Comi, & Filippi, 2007). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 24 Sormani et al.s (2007) study included 539 PwMS from the placebo group of a clinical trial. The univariate Cox analysis that produced variables with a p value less than .20 were used in the multivariate Cox analysis and time to first relapse was the dependent variable (Sormani et al., 2007). Variables that had a p value of .01 were retained in the final model. The final model produced a linear predictive score that was calculated using the variables included in the model and their estimated coefficients (Sormani et al., 2007). Creating a linear predictive score using a multivariate Cox analysis is a traditional statistical method. The study lacks comparison to other traditional and machine learning algorithms to evaluate fit and performance. In addition, the final model has only two independent predictors of relapse, the number of enhancing lesions on a baseline MRI and the number of relapses during the previous two years (Sormani et al., 2007). The number of features included in the final model may need to be expanded to generate a more accurate predictive model. There is a lack of research on predicting MS hospital utilization, specifically MS allcause UC. A recent study forecasted MS hospitalization in the U.S. from 2017-2040, and utilized an Autoregressive Integrated Moving Average (ARIMA; Sharma, Bittner, & Cho, 2019). Results showed hospital admissions were predicted to increase by 32% in 2030 (Sharma et al., 2019). However, this study does not compare with other sophisticated predictive models, and does not predict all-cause UC utilization. In 2005, a cross-sectional study of 4,000 MS participants was done to estimate current costs and quality of life of participants treated with disease-modifying drugs (DMDs) in the U.S, and an overall assessment of MS cost (Kobelt, Berg, Atherly, & Hadjimichael, 2006). Results showed that 28.8% of participants had a relapse during the past three months and total average costs were estimated at $47,215 per patient per year (Kobelt et al., 2006). Iezzoni and Ngo (2007) completed a study and interviewed 983 MS working-age STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 25 individuals. Results of the study showed significant impact on cost of MS care and disability impact. Of those interviewed, 27.4% indicated that, their health insurance concerns had significantly affected employment decisions since being diagnosed with MS (Iezzoni & Ngo, 2007). Furthermore, 27.4% put off or postponed seeking needed health care because of costs, 16.4% reported difficulty paying for health care, and 22.3% delayed filling prescriptions, skipped medication doses, or split pills because of costs (Iezzoni & Ngo, 2007). In addition, 26.6% reported significant concerns about affording food, utilities, and housing (Iezzoni & Ngo, 2007). Overall, there has been limited research on predicting MS relapses and predicting MS allcause UC utilization to date. These studies used traditional statistical methods with no emphasis on fit or performance of the model. In addition, all of the statistical methods used were traditional in nature and did not use modern predictive analytics techniques such as random forest, ridge or LASSO regression which will likely perform better and provide more robust and accurate results. Statistical modeling comparisons. The main purpose of this research study is to compare statistical model classes between traditional methods and more sophisticated predictive analytics. There is some research suggesting what are the best model comparison approaches and how do they compare. Couronn, Probst, and Boulesteix (2018) conducted a model comparison paper between random forest and logistic regression. This was a large-scale benchmarking experiment based on 243 datasets comparing the prediction performance of random forest with logistic regression (Couronn et al., 2018). This model comparison used AIC, area under the curve (AUC), and Brier score and showed the delta between logistic regression and random forest. Results of the study showed that random forest performed more favorably over logistic regression in 67 of the datasets analyzed (Couronn et al., 2018). Mean differences STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 26 between logistic regression and random forest was .03 for accuracy, -.03 for Brier score, and .04 for AUC. Couronn et al.s (2018) comparison study was utilized as a framework for this proposed study. Zhang, Lu, and Hou (2019) conducted research on a model comparison between random forest and logistic regression in predicting diabetes. The study calculated AUC, sensitivity and accuracy for model comparison. Results showed that the accuracy of random forest was slightly higher than logistic regression (0.87 versus 0.86; Zhang et al., 2019). The AUC was similar between random forest and logistic regression and random forest (0.94 vs. 0.93; Zhang et al., 2019). This study is limited as the researchers only compared a few different classification measures and excluded measures such as precision, F1 Score, and Matthews correlation coefficient (MCC) which this study has built upon (Zhang et al., 2019). Matthews correlation coefficient was originally developed in 1975 for comparison of chemical structures (Matthews, 1975). The MCC is similar to computing any correlation coefficient, but for two binary variables, true class and predicted class (Baldi, Brunak, Chauvin, Andersen, & Nielsen, 2000). The MCC is always between -1 and +1, a value of -1 denotes total disagreement and +1 total agreement (Baldi et al., 2000). Baldi et al. (2000), re-proposed Matthews correlation coefficient as the standard performance metric for machine learning. Chicco and Jurmans (2020) study utilized six synthetic use cases in different genomics scenario, and showed MCC produced a more informative and accurate score in assessing binary classification models, than accuracy and F1 score. The F1 score is defined as the harmonic mean of precision and recall (Chicco & Jurman, 2020). The study suggests that the MCC should be preferred to F1 and accuracy when evaluating binary classification tasks by scientific communities (Chicco & Jurman, 2020). When a dataset is unbalanced, accuracy is not a good STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 27 measurement because it produces an over optimistic estimation of the classifier ability on the majority class, but MCC performs better in this situation (Chicco & Jurman, 2020). Thus, the MCC is a reliable statistical measure and produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (Chicco & Jurman, 2020). The confusion matrix is a critical tool used to determine the models accuracy, is part of the models results, and contains four metrics (true positives, false negatives, true negatives, and false positives). There is a tradeoff between prediction accuracy and model interpretability when comparing parametric and nonparametric models in statistical learning methods. More restrictive models are generally more interpretable but sometimes less accurate in prediction (e.g., linear or logistic regression; James et al., 2013). If inference is the goal, then more restrictive models such as linear regression are usually preferred. Non-parametric methods such as decision trees and support vector machines are usually less flexible models and are less interpretable, but sometimes more accurate (James et al., 2013). With regards to this study, logistic, LASSO, and ridge are parametric methods and random forest is nonparametric. Figure 2 shows the flexibility of a method increases as the interpretability decreases (James et al., 2013). In addition, machine learning algorithms have the ability to include many features and perform optimally, where logistic regression may take more time to engineer and optimize the model. Current research on relapse and all-cause UC does not enable broad and modern use of predictive modeling techniques. In addition, there is no research comparing fit and performance between predictive analytics and statistical methods. These limitations present challenges for cost-effectiveness analyses and the overall assessment of the quality and cost of MS healthcare services delivery. These studies have made the advancement of analysis of MS at the population STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 28 level. However, they have not conducted systems-level analysis nor investigated predictive analytics techniques concerning MS relapse or all-cause UC. There are healthcare quality outcomes and cost implications for this work beyond the methodological comparisons of statistical models in theoretical research. Predictive analytics methods have the potential to empower and accelerate the capability of LHS to predict and respond to emerging health needs of people with costly, complex, chronic 3C conditions like MS and better inform continuous improvement efforts. This could potentially result in achieving more efficient and effective healthcare outcomes and saving time and overall cost. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 29 Chapter 2 Method Parent Study Design The parent research study, MS-CQI (Appendix A), utilizes a prospective step-wedge randomized design. Compared to a standard randomized clinical trial or cluster-randomized design, the step-wedge design can allow for additional comparisons within each MS center in a pre-clinical and post-clinical quality intervention (Rapkin et al., 2012). Randomized clinical trials are the gold standard in research design for inferring a causal relationship (Cartwright, 2010). MS-CQI is a two-part, 3-year clinical quality improvement prospective study that started in June 2017 and concluded in June 2020. There are two levels of participation in this study: (a) system-level administrative; and (b) individual-level clinical Patient Reported Outcomes (PROs). In MS-CQI, the term system level refers to the MS clinical sites and population level refers to the whole MS collaborative which consists of all four clinical sites aggregated together. Individual level data are available within each MS clinical center. These data created a combined MS clinical quality intervention (MS-CQI) database. Benchmarking and analysis of data from each center has been conducted via the MS-CQI database quarterly throughout the three-year study. Baseline data were collected during the first year of the study. In the second part of the study, years two and three, investigations were conducted on the effect of a clinical quality intervention on primary endpoints and selected secondary measures at the system and individual level outcomes. Protected health information (PHI) was collected for the individual level PRO portion of the study; therefore, written informed consent was obtained for participation in this part of the study. (Oliver et al., 2019). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 30 Proposed Study Design This study was built on the standard MS-CQI protocol which has established the MSCQI systems-level database and utilized data from the parent study for secondary analyses. An analytic comparison study was proposed based on the third year of data, June 2019-June 2020. The MS-CQI team has studied system-level variation in relapses and all-cause UC for individual sites, between sites, and for MS-CQI collectively as part of the parent study. The proposed study predicted relapse and all-cause UC and compares these outcomes through various types of statistical models. Data collected across four clinical MS centers include administrative data and eight clinical electronic health record (EHR) clinical outcome measures. These outcome measures are described in the subsequent section. The MS-CQI study is using traditional methods of analysis, and this study builds upon that by studying newer approaches and comparing these with the standard approaches. Logistic regression and maximum likelihood estimation methods were used for standard inferential analyses. Ridge, LASSO, and random forest was used for predictive analytics. Participants This study used EHR data extracted at the individual level and rolled up to the system and population level. Inclusion criteria included participants age 18 years or older, with MS presenting to any of the four centers, who entered the study in any quarter (7/1/2017-6/30/2020). Exclusion criteria included cases with missing or incorrectly input data and those who refused to participate in the study. For the purpose of this study, data collected in year 3 was utilized. Additional methods for participants are available in the parent study (See Appendix A). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 31 Setting Methods for setting are described by the parent study protocol for MS-CQI (see Appendix A). Data Management Data were extracted in a de-identified form via data downloads from electronic medical records (EMRs) and administrative records from participating centers. Each MS-CQI clinical center enters individual level data, and the data was managed using REDcap (Research Electronic Data Capture) electronic data capture tools hosted at Dartmouth. REDCap is a secure, web-based software platform designed to support data capture for research studies, providing (a) an intuitive interface for validated data capture; (b) audit trails for tracking data manipulation and export procedures; (c) automated export procedures for seamless data downloads to common statistical packages; and (d) procedures for data integration and interoperability with external sources (Harris et al., 2019; Harris et al., 2009). No PHI was entered into this database. All PHI remained at specific clinical center locations. All data were initially managed in the REDcap database, and subsequently extracted into SAS 9.4 for data management, and R for statistical analysis. Additional methods for data management were established by the MS-CQI parent study (See Appendix A). Procedures Recruitment. Methods for recruitment were established by the MS-CQI parent study (See Appendix A). Informed consent. The institutional review board (IRB) at Dartmouth (Committee for Protection of Human Subjects at Dartmouth College) approved the parent MS-CQI study, and a reliance agreement between Dartmouth and the Human Research Protection Program at the STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 32 University of Indianapolis was put into place before data analysis was conducted. Additional methods for informed consent were established by the MS-CQI parent study (See Appendix A). Continuous Quality Improvement Intervention The clinical quality improvement intervention employed in the parent study is an IHI Breakthrough Series CQI intervention, including an improvement collaborative and professional improvement coaching (Oliver et al., 2019). Methods for CQI intervention were established by the MS-CQI parent study (See Appendix A). Data and Data Collection for Relapses and All-Cause Hospitalization Each of the four MS-CQI clinical centers abstracted participant data from their own EHRs. The data were deidentified and then input into REDcap. The following is an estimated sample size for unique participants by center for EHR year 3: Center A: n = 977; Center B: n = 460; Center C: n = 539; Center D: n = 556. Total sample size for year 3 is 2,532. For this study, data was extracted from REDcap. Data management for this study used SAS/STAT software, Version 9.4 (Copyright 2019; SAS Institute Inc., Cary, NC). Analysis for this study was conducted in R (R Core Team, 2014). The following variables were extracted and included in the predictive models. Patient level predictors include: Age is a continuous variable measured in years Sex is a categorical dichotomous variable (male = 0, female =1) MS Center is an ordered categorical variable (1, 2, 3, 4) MS phenotype is an ordered categorical variable (RRMS, SPMS, PPMS, PRMS, other). Clinical level predictors include: STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Participants on at least one DMT is a dichotomous variable (0 = no, 1 = yes) Participants with at least one brain MRI is a dichotomous variable (0 = no, 1 = 33 yes) Participants with at least one thoracic MRI is a dichotomous variable (0 = no, 1 = yes) Participants with at least one cervical MRI is a dichotomous variable (0 = no, 1 = yes) Participants with at least one MS relapse is a dichotomous variable (0 = no, 1 = yes) Participants with at least one hospitalization is a dichotomous variable (0 = no, 1 = yes) Participants with at least one ED visit is a dichotomous variable (0 = no, 1 = yes) Participants with at least one UC visit is a dichotomous variable (0 = no, 1 = yes) Exploratory Data Analysis Descriptive statistics include frequency distributions for categorical variables and means with standard deviations for normally distributed continuous variables. The output of the descriptive statistics was reported in tabular form for each center and overall. Appropriate power tests were conducted to ensure at least a .80 level, and demographic data were analyzed for differences across centers. Mean differences were tested via one-way ANOVA (analysis of variance) for continuous variables and chi-square for categorical variables with a significance level of = .05. Bivariate analysis was conducted to determine associations between pairs of variables and which variables should be included in the predictive models (p < .10). If there were missing data, an average imputation was computed, and the data were included in the analysis. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 34 Boxplots were utilized to assess outliers in continuous data and frequency tables for categorical data. If extreme observations were detected, the analysis continued to better understand if the outliers impacted the analysis results. If the results were affected by the outliers, those observations were removed from the dataset. Predictive Models Logistic regression was utilized as the traditional model because it is often used for classification predictive models in healthcare. Its popularity in health sciences is due to the ability of logistic regression to give a discrete dichotomous outcome (e.g., disease or no disease; Tabachnick, Fidell, & Ullman, 2019). In the proposed study, logistic regression was compared with other machine learning models, specifically ridge, LASSO, and random forest. The area under the curve of these models was compared to assess which model had the best goodness of fit to suggest which approach is best at predicting MS relapse and MS all cause-UC utilization. In addition, accuracy, true positive rate (TPR), false-positive rate (FPR), negative predictive value (NPV), precision (PPV), F1 measure, and MCC was compared across models. Each model was separately estimated for each outcome, and model results were tabled and compared. Logistic regression. Logistic regression modeling allows a discrete outcome variable (dependent variable) to be modeled as a function of other variables that can be discrete, continuous, dichotomous, or a mix (Tabachnick et al., 2019). Logistic regression analysis is also used when the distribution of responses on the dependent variable is likely to be nonlinear (Tabachnick et al., 2019). The logistic regression is a nonlinear model and the equations utilized to describe the outcomes are more complex than those for multiple regression (Tabachnick et al., 2019). However, this nonlinear function is based on the best linear combination of predictors (Tabachnick et al., 2019). This linear equation uses the logit function as our link function and is STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 35 the natural (loge) of the probability of being in one group divided by the probability of being in the other group (Tabachnick et al., 2019). The technique for estimating coefficients in logistic regression is maximum likelihood. The purpose of this procedure is to find the best linear combination of predictors to maximize the likelihood of obtaining the observed outcome frequencies (Tabachnick et al., 2019). Since the logit function is utilized, the output is the logodds; thus, we call this model logistic regression. The range of log-odds is from to . In this study the logistic model was optimized initially through the bivariate analysis in exploratory data analysis (EDA). In the modeling process the variables that were selected through the EDA were run through backward elimination at the statistical significance of p < .05. This process engineered and optimized the logistic regression to compete equitably with the machine learning algorithms. Ridge regression and LASSO. Ridge regression and LASSO are both shrinkage methods and can be used with many features in the model without overfitting (Hastie, Tibshirani, & Friedman, 2009; James et al., 2013). Both of these models shrink or regularize the coefficient estimates (shrinks them) towards zero (Hastie et al., 2009; James et al., 2013). By shrinking coefficient estimates, this improves the fit of the model by reducing the models overall variance (Hastie et al., 2009; James et al., 2013). Ridge and LASSO work by adding a penalty term to the log-likelihood function. For ridge regression, the penalty term is 2 , and for LASSO, it is |j|. (Hastie et al., 2009; James et al., 2013). The penalty term is a shrinkage penalty, and when the beta estimates are small it has the effect of shrinking the parameters close to zero (James et al., 2013). The tuning parameter controls the relative impact of the shrinkage penalty on the coefficient estimates. When the tuning parameter, , is positive, the coefficient estimates of large values of are STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 36 shrunk towards zero. When = 0, the penalty term has no effect, and the ridge regression will produce least squares estimates (Hastie et al., 2009; James et al., 2013). The value of is critical and can be chosen via cross-validation (James et al., 2013). Cross-validation to be discussed subsequently. Ridge will never force any variables to zero, so every variable is retained in the final model, which affects the interpretability of the model (Hastie et al., 2009; James et al., 2013). The penalty term makes the ridge regression model have decreased variance but increased bias (Hastie et al., 2009; James et al., 2013). In this circumstance, bias refers to how far off our predicted value is compared to the actual value. Variation is the spread or variation in the predicted values. The key in machine learning is to find a balance between bias and variation. For example, in machine learning the model is often complex, as complexity increases, variance increases and bias decreases. Finding the perfect balance between bias and variance increases the accuracy and generalizability of the model. Figure 3 shows the ridge regression coefficient estimates for each value of . As increases, the coefficient estimates approach zero (the horizontal line) and reduces the complexity of the model (James et al., 2013). LASSO works similarly to ridge but has an alternative penalty term of |j|. The LASSO penalty term will set some variables to zero that are not important to the outcome; this occurs when the tuning parameter, , is sufficiently large and therefore performs variable selection (James et al., 2013). When = 0, the resulting model is an ordinary least squares (OLS) model. When is very large, the resulting model is the null model (intercept-only model) as all variables are forced to 0 (James et al., 2013). LASSO will produce a model that is easy to interpret due to variable selection and with high predictive power (Hastie et al., 2009; James et al., 2013). Ridge STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 37 regression and LASSO are similar in that as the tuning parameter increases, so does bias, but variance decreases (Hastie et al., 2009; James et al., 2013). Random forest. Decision trees are used frequently because they are similar to the way human beings make decisions and thus are easier to explain and visualize (James et al., 2013). However, their predictive accuracy may not be as strong as other machine learning predictive models (James et al., 2013). Random forest is a decision tree-based algorithm that does not have overfitting issues as most decision-tree algorithms do, and increased predictive accuracy (Hastie et al., 2009; James et al., 2013). This is because random forest is different from other decision trees; it decorrelates the trees (Hastie et al., 2009; James et al., 2013). The decorrelation process is also called bagging, which generates new training data sets from an original data via random sampling with replacement, also known as bootstrapping. This tree model uses various sets of training data, but each time there is a split in the tree a random sample of predictors are chosen from the full set of predictors (Hastie et al., 2009; James et al., 2013). Each individual bootstrapped data set is then used to construct a tree within the forest. This methodology reduces overfitting the model and is far less likely than other decision tree models to have a small set of strong predictors in all the tree splits (Hastie et al., 2009; James et al., 2013). Cross-validation. Resampling methods are an invaluable tool in the era of modern statistics (James et al., 2013). Resampling involves drawing samples repeatedly from the training set and refitting the model on each of the given samples to learn additional information on the fitted model (James et al., 2013). This approach allows us to learn new information about the model by fitting it multiple times and comparing different sample results as opposed to just fitting it once in the training sample (James et al., 2013). Data for the MS-CQI study are highly unbalanced-- particularly the relapse outcome. The low prevalence causes an imbalanced dataset STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 38 as there are far many more non-events (no relapse) than events (relapse). During crossvalidation, a synthetic minority over-sampling technique (SMOTE) algorithm was used to help with the imbalanced nature of the data. This algorithm is well known and is used to synthetically generate new examples of the minority class via nearest neighbors and the majority class, in this case (no-relapses) are also under-sampled, for a more balanced dataset. (Chawla, Bowyer, Hall, & Kegelmeyer, 2002). Ridge, LASSO, and random forest methods generate a variable importance plot, which is scored from 0 to 100 and gives a value based on the statistical significance and effect on the model Cross-validation is a type of resampling method that is used to estimate the test error of the statistical method, which gives information on model performance or flexibility (James et al., 2013). Cross-validation can be a useful approach in classification models where the outcome Y is qualitative (James et al., 2013). In this situation, the cross-validation uses the number of misclassified observations (James et al., 2013). For this study k-fold cross-validation was used as the resampling method, where k = 10. The k-fold approach randomly divides the observations in the data set into folds or k groups of equal size (Hastie et al., 2009; James et al., 2013). The initial fold is the validation set and then the method is fit on the remaining k-1 folds (Hastie et al., 2009; James et al., 2013). This process is then repeated k times with a different set of data and each time is treated as a validation set (James et al., 2013). The standard approach for classification problems in data science is 10-fold cross-validation. For this study, the best model from each cross-validation training of 10 models is selected as the champion (optimized) model. During cross-validation each model was separately optimized for internal tuning using accuracy, MCC, and F1. For the endpoint of relapse, logistic regression, LASSO, ridge and random forest were each cross-validated ten times by three STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 39 different optimizations for a total of 12 champion models that came from a total of 120 models. For the endpoint of all-cause UC logistic regression, LASSO, ridge and random forest each was cross-validated ten times by two different optimizations (accuracy and MCC) for a total of eight champion models that came from a total of 80 models. The additional optimization method of F1 was not needed for all-cause UC as the dataset was already sufficiently balanced and produced similar results for accuracy and MCC. Model comparison. The AUC, accuracy, TPR, FPR, precision, NPV, F1 score, and MCC was computed for all models. In statistical tests with binary outcomes, the accuracy is evaluated by sensitivity and specificity. Sensitivity also known as recall is defined as true positives, and specificity is defined as true negatives. Plotting the sensitivity versus 1- specificity generates the receiver operating characteristic (ROC) curve (Hajian-Tilaki, 2013). The AUC is an effective measure of this accuracy in predictive models (Hajian-Tilaki, 2013). Generally, the AUC range is between .50 and 1.0, and is also known as the c-statistic. An AUC of .50 indicates a prediction by chance, and AUC of 1.0 indicates a perfect prediction (Tabachnick et al., 2019). Accuracy is an overall average of how well the model predicts as well as its computational simplicity. In data science positive predictive value (PPV) and precision have the same mathematical definition, however the actual nomenclature is different. Precision or PPV is defined as true positives divided by true positives plus false positives. The precision or PPV defines the probability of having the disease or state in an individual with positive result (imundi, 2009). The F1 score is defined as the harmonic mean of precision and recall (Chicco & Jurman, 2020). The MMC is not affected by the unbalanced datasets, and is a contingency matrix method of calculating the Pearson product- moment correlation coefficient between actual and predicted values (Chicco, & Jurman, 2020). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 40 Model metrics in Table 1 and Table 2 were calculated for all models in this study and ridge, LASSO, and random forest were compared to logistic regression. Building upon the framework established by Couronn and colleagues (2018), the delta between AUC, MCC, and F1 Score were added and compared for each model pair. This combination of measures was chosen based on model comparison studies in the literature, as well as MCC being the new measure to evaluate machine learning models performance and fit (Chicco & Jurman, 2020; Couronn et al., 2018; Zhang et al., 2019). Historically, F1 score and AUC were standards in comparing model performance and fit. Indices were created and differences were calculated as follows: perfA=perfRF perfLR perfA1=perfLASSOperfLR perfA2=perfRRperfLR perfM=perfRFperfLR perfM1=perfLASSOperfLR perfM2=perfRRperfLR perfF=perfRFperfLR perfF1=perfLASSOperfLR perfF2=perfRRperfLR Before beginning the analysis and to evaluate internal validity, a training and test data set were split into 70% and 30% test for validation. All models in this study followed the same validation method. Unless noted, the threshold for statistical significance is p < .05. All models have the same dataset, predictors, and outcomes, and all models were trained and fit on the entire data set of the backwards logistic regression model for consistency. Several data visualizations STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 41 were implemented as part of this study, including ROC and gain curves. Gain curves were used to visually evaluate model performance in a binary predictive model. The visualization presents the percentage of captured positive responses as a function of a selected percentage of the sample. The following R packages were used in this study: tidyverse: package for data import caret: package for modeling and machine learning yardstick: package used to quantify model fit and performance glmnet: package for generalized linear models broom: package for data manipulation and analysis dplyr: package for data transformation and manipulation ggplot2: package for data visualization STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 42 Chapter 3 Results Center and Participant Characteristics Characteristics of participating MS-CQI centers are given in Table 1 for Year 3 of the study. The four participating centers (1-4), represent varying contexts ranging from urban academic MS centers to a private practice setting. Only two participants were dropped due to missing data. A total of 2,532 unique PwMS were followed by MS-CQI in the third year of the study, and volume varied substantively across centers from a low of 460 in center 2 to a high of 977 in center 1. Overall, the general demographic characteristics of the study population align with those established in general MS populations, including a majority of female gender (76%), and relapsing MS subtype (81%). Mean age was 50 years (see Figure 4). A number of important characteristics varied significantly (p < .001) across centers, including MS diagnosis type, age, and gender. Clinical Outcomes and Center Level Variation Chi-square tests identified numerous differences in center level performance outcomes (Table 2). Approximately 73% of participants were on disease modifying therapy (DMT), and center-specific performance ranged from 60% to 95% (p < .001). Overall brain (head) MRI utilization was 48% with variation observed in center level performance (37-56%, p = .051). Overall cervical (upper neck) and thoracic (middle back) MRI utilization was much lower (27% and 15%, respectively), with significant center level variation again observed in the cervical range: 18-35%, (p < .001); thoracic range: 4-23%, (p < .001). Approximately 4% of participants experienced an MS relapse and center-specific performance ranged from 2.6% to 6.5% (p < .001). The ANOVA procedure was used to identify significant differences on annualized rates STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 43 for ED and UC Utilization, relapse rate, and age across centers. All the measures varied significantly (p < .001; see Table 2). The proportion of PwMS with at least one episode of acute care utilization (UC, ED, and hospitalizations) were relatively low: UC = 2.7%, ED = 8.2%, and hospitalization = 7.0%. Significant center level variation in performance was observed here as well. The proportions of PwMS with at least one hospitalization ranged from a high of 10.7% to a low of 4.3% (p < .001), proportions of PwMS with at least one ED visit ranged from a high of 11.1% to a low of 5.8% (p < .001), and proportions of PwMS with at least one UC visit ranged from a high of 5.2% to a low of 0.2% (p < .001). Utilization in these categories was also calculated in terms of utilization rates, and demonstrated statistically significant variation (p < .001) across centers (see Table 2). A Spearman correlation showed .98 - .99 positive correlation between clinical level dichotomous variables of ED, UC, hospitalization and relapse and the associated continuous level utilization variables. Therefore, the continuous level utilization variables were removed from any further modeling and analysis. Training and Testing Datasets The main dataset was spilt into a 70% training dataset that included 1,773 participants and a test dataset that included 30% of the cohort or 759 participants. Detailed information on the training and testing cohorts are included in Table 3. Bivariate analysis for determining predictors included in the predictive models is available in Table 4. Quantitative Aim 1: Relapse Predictive Model Comparison The first aim of the study was to compare logistic regression to LASSO, ridge and random forest in predicting MS relapse. Relapse has very low prevalence. There was a total of 97 relapses in the dataset. The training data set had 74 relapses or 4.2% and the test dataset had 23 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 44 or 3.1% relapses. The low prevalence of relapse causes an imbalanced dataset, as there are far many more non-events than events. An initial backwards logistic regression was fit on the full dataset to set up the training and fit for consistency on all models. Based on the bivariate analysis and full dataset results the following features were included in all relapse models: age, gender, MS center, MS phenotype, DMT, ED visit, UC visit, brain MRI, cervical MRI, and thoracic MRI. Relapse was set as the target variable (outcome). There were 120 models run via crossvalidation for three different optimizations (accuracy, MCC, F1) and four different models (logistic, LASSO, ridge, random forest) within each paradigm. Twelve champion models were selected for the endpoint (relapse). Logistic regression (accuracy). A logistic regression using backward elimination was conducted controlling for confounding effects of age, gender and center. Center, MS phenotype, ED visit, brain MRI, thoracic MRI were significant at the p < .05 level. UC visit was not significant at p = .07. The confusion matrix predicted 582 true negatives, 18 false positives, 148 false negatives and 11 true positives. Some key measures of the model were accuracy = 0.78, AUC = 0.63, TPR = 0.38, FPR = 0.80, MCC = 0.08 and F Score was 0.12. Additional model performance measures are in Table 6. LASSO (accuracy). A LASSO was conducted controlling for confounding effects of age, gender and center. The model produced an intercept only model due to shrinking the rest of the parameter estimates to zero. The confusion matrix predicted 730 true negatives, 29 false positives, 0 false negatives and 0 true positives. The model predicted no relapses on all observations, which is why certain performance measures such as MCC and F1 score could not be calculated. Some key measures of the model were accuracy = 0.96, AUC = 0.50, TPR = 0, and FPR = 1. Additional model performance measures are in Table 7. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 45 Ridge (accuracy). A ridge regression was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: thoracic MRI (100.0), brain MRI (84.0), center 4 (82.9), UC visit (69.5), RRMS (60.5), SPMS (59.8), DMT (57.6), cervical MRI (54.5), ED visit (52.7), center 2 (50.5), center 3 (38.6), other phenotype (20.8), gender (4.5), Age (0.77) and PRMS (0). The confusion matrix predicted 730 true negatives, 29 false positives, 0 false negatives and 0 true positives. The model predicted no relapses on all observations, which is why certain performance measures such as MCC and F1 score could not be calculated. Some key measures of the model were: accuracy = 0.96, AUC = 0.67, TPR = 0, and FPR = 1. Additional model performance measures are in Table 8. Random forest (accuracy). A random forest was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: thoracic MRI (100), center 4 (83.0), brain MRI (69.0), DMT (40.1), cervical MRI (39.5), ED visit, (35.2), gender (35.0), center 2 (34.7), center 3 (27.9), RRMS (26.9), Age (18.5 years), other phenotype (14.4), SPMS (13.3), UC visit (3.3) and PRMS (0). The confusion matrix predicted 677 true negatives, 20 false positives, 53 false negatives and 9 true positives. The model predicted no relapses on all observations. Some key measures of the model were: accuracy = 0.90, AUC = 0.69, TPR = 0.31, FPR = 0.93, MCC = 0.17 and F1 Score was 0.20. Additional model performance measures are in Table 9. All relapse models (accuracy). Evaluating models for AUC, random forest performed the best at (0.69), ridge (0.67), logistic regression (0.63), and LASSO (0.51) (see Figure 5). Random forest AUC was 9.5% higher than logistic regression. LASSO and ridge models predicted no relapses for all observations therefore; MCC and F1 Score were not calculated. MCC for random forest was 0.17 versus 0.08 for logistic, a 100% difference. The F1 score was STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 46 0.198 for random forest versus 0.12 for logistic regression, a 69% difference. Figure 6 shows the gain curves across models and upon visual inspection random forest and ridge models need to test about 25% of the sample population to capture 50% of positive responses however logistic requires 35% and LASSO 50%. Additional model performance measures are in Table 22. LASSO (MCC). A LASSO regression was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: thoracic MRI (100.0), center 4 (79.0), UC visit (61.5), RRMS (42.2), brain MRI (39.6), center 2 (36.6), gender (25.5), ED visit (22.1), center 3 (21.2), other phenotype (7.0), Age (0.72), cervical MRI (0), SPMS (0), DMT (0) and PRMS (0). The confusion matrix predicted 575 true negatives, 17 false positives, 155 false negatives and 12 true positives. Some key measures of the model were accuracy = 0.77, AUC = 0.62, TPR = 0.41, FPR = 0.79, MCC = 0.09 and F1 score was 0.12. Additional model performance measures are in Table 10. Ridge (MCC). A ridge regression was conducted controlling for confounding effects of age, gender, and center. The models variable importance was as follows: PRMS (100.0), thoracic MRI (98.2), UC visit (78.4), center 4 (75.4), brain MRI (67.3), center 3 (52.0), ED visit (49.9), SPMS (46.5), cervical MRI (45.4), center 2 (44.4), RRMS (43.50), DMT (37.2), gender (21.3), other phenotype (15.4), and age (0). The confusion matrix predicted 603 true negatives, 17 false positives, 127 false negatives and 12 true positives. Some key measures of the model were accuracy = 0.81, AUC = 0.65, TPR = 0.41, FPR = 0.83, MCC = 0.12 and F1 Score was 0.14. Additional model performance measures are in Table 11. Random forest (MCC). A random forest was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: thoracic MRI (100.0), center 4 (98.0), brain MRI (67.6), ED visit, (43.8), DMT (41.1), center 2 (33.1), RRMS STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 47 (32.0), Age (31.5), gender (27.3), cervical MRI (26.2), center 3 (25.5), SPMS (20.5), other phenotype (10.7), UC visit (1.7) and PRMS (0). The confusion matrix predicted 669 true negatives, 19 false positives, 61 false negatives and 10 true positives. Some key measures of the model were accuracy = 0.90, AUC = 0.69, TPR = 0.35, FPR = 0.92, MCC = 0.17 and F1 Score was 0.20. Additional model performance measures are in Table 12. All relapse models (MCC). For the MCC optimization, logistic regression results were the same as accuracy. Evaluating models for AUC, random forest performed the best at (0.69), ridge (0.65), logistic regression (0.63), and LASSO (0.62) (See Figure 7). Random forest AUC was 9.5% higher than logistic regression. MCC for random forest was (0.17), ridge (0.12), LASSO (0.09) and (0.08) for logistic regression. The F1 score for random forest was (0.20), ridge (0.14), LASSO (0.12) and (0.12) for logistic regression. Random forest outperformed logistic regression on MCC and F1 score by 107% and 71% respectively. LASSO and ridge also outperformed logistic regression. Figure 8 shows the gain curves across models and upon visual inspection all models need to test about 30-35% of the sample population to capture 50% of positive responses. Additional model performance measures are in Table 22. LASSO (F1). A LASSO regression was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: thoracic MRI (100) the rest of the features had values of 0. The confusion matrix predicted 730 true negatives, 29 false positives, 0 false negatives and 0 true positives. Some key measures of the model were accuracy = 0.96, AUC = 0.59, TPR = 0, and FPR = 1. The model predicted no relapses on all observations, which is why certain performance measures such as MCC and F1 score could not be calculated. Additional model performance measures are in Table 13. Ridge (F1). A ridge regression was conducted controlling for confounding effects of age, STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 48 gender and center. The models variable importance was as follows: thoracic MRI (100), center 4 (92.6), brain MRI (68.4), UC visit (67.6), DMT (58.2), center 2 (54.5), cervical MRI (51.1), center 3 (47.8), RRMS (45.3), SPMS (40.5), ED visit (23.6), gender (13.2), other phenotype (4.6), Age (1.2) and PRMS (0.0). The confusion matrix predicted 654 true negatives, 17 false positives, 76 false negatives and 12 true positives. Some key measures of the model were accuracy = 0.88, AUC = 0.65, TPR = 0.41, FPR = 0.90, MCC = 0.19 and F1 score was 0.21. Additional model performance measures are in Table 14. Random forest (F1). A random forest was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: center 4 (100.0), thoracic MRI (77.9), brain MRI (53.6), Age (42.4), ED visit, (36.6), cervical MRI (32.9), DMT (17.4), gender (15.3), center 2 (13.4), center 3 (7.4), SPMS (5.5), other phenotype (4.97), PRMS (1.19), UC visit (1.16) and RRMS (0). The confusion matrix predicted 645 true negatives, 23 false positives, 85 false negatives and 6 true positives. Some key measures of the model were accuracy = 0.86, AUC = 0.67, TPR = 0.21, FPR = 0.88, MCC = 0.05 and F1 score was 0.10. Additional model performance measures are Table 15. All relapse models (F1). For the F1 optimization, logistic regression results were the same as accuracy. Evaluating models for AUC, random forest performed the best at (0.67), ridge (0.65), logistic regression (0.63), and LASSO (0.59), but not dramatically different than logistic regression (See Figure 9). Ridge AUC was 6.3 % higher than logistic regression. MCC for ridge was (0.19), logistic regression (0.08) and random forest (0.05). The F1 score for ridge was (0.21), logistic regression (0.12), and random forest was (0.10). Ridge outperformed logistic regression on MCC and F1 score by 122% and 75% respectively. Logistic regression outperformed random forest on MCC and F1 measure. Figure 10 shows the gain curves across STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 49 models and upon visual inspection all models need to test about 35-37% of the sample population to capture 50% of positive responses. Additional model performance measures are in Table 22. Relapse indices. The delta between AUC, MCC, and F1 scores were added and compared for each model pair. This combination of measures was chosen based on model comparison studies in the literature evaluating machine learning models performance and fit and building upon those studies (Chicco & Jurman, 2020; Couronn et al., 2018; Zhang et al., 2019). Comparing relapse indices across models random forest significant outperformed logistic regression and other machine learning algorithms regardless of internal cross-validation optimization. For perfA there was a 0.02 or 27.1% difference between random forest and logistic regression for accuracy optimization and perfM a 27.5% difference. However, for perfF, which was the F1 optimization, logistic regression and random forest performed relatively the same (0.82 vs. 0.83). The perfM1 and perfM2 (MCC optimization) LASSO and ridge outperformed logistic regression (0.9%, 9.4%) respectively. The perfF2 index showed that ridge outperformed logistic by 25.8%. The perfA1, perfA2, and perfF1 could not be calculated due to LASSO and ridge models predicted no relapses on all observations, therefore, certain performance measures such as MCC and F1 score could not be calculated. See table 24 and figures 15 and 16 for comprehensive data tabulations and visualizations. Quantitative Aim 2: All-Cause Utilization Predictive Model Comparison The second aim of the study is to compare logistic regression to LASSO, ridge and random forest in predicting MS all-cause utilization. MS all-cause utilization included all-cause UC visits plus all-cause ED visits. All-cause UC has low prevalence, for year three there was a total of 258 all-cause UC visits or 10.2% of the total population. The training data set had 181 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 50 all-cause UC visits or 10.2% and the test dataset had 77 or 10.2% all-cause UC visits. The low prevalence causes an imbalanced dataset, as there are far many more non-events (no all-cause UC) than events (all-cause UC). An initial backwards logistic regression was fit on the full dataset to set up the training and fit for consistency on all models. Based on the bivariate analysis and full dataset results the following features were included in all relapse models: age, gender, MS center, MS phenotype, DMT, hospitalization, relapse, brain MRI, cervical MRI, thoracic MRI, and all-cause UC as the target variable (outcome). There were 80 models run via crossvalidation, for two different optimizations (accuracy, MCC) and 4 different models (logistic, LASSO, ridge, random forest) within each paradigm and eight champion models selected for the endpoint of all-cause UC. There are at total of 80 models and 8 champion models in this aim since the F1 optimization was not included as part of the modeling process. For this endpoint, the F1 results were similar to the other optimization techniques (accuracy, MCC). Logistic regression (accuracy). A logistic regression using backward elimination was conducted controlling for confounding effects of age, gender and center. Center, MS phenotype, hospitalization, and thoracic MRI were significant at the p < .05 level. Relapse visit was fairly significant at p = .051. The confusion matrix predicted 647 true negatives, 43 false positives, 35 false negatives and 34 true positives. Some key measures of the model were accuracy = 0.90, AUC = 0.76, TPR = 0.44, FPR = 0.95, MCC = 0.41, and F score = 0.47. Additional model performance measures are in Table 16. LASSO (accuracy). A LASSO was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: hospitalization (100) the rest of the features had values of 0. The confusion matrix predicted 660 true negatives, 45 false positives, 22 false negatives and 32 true positives. Some key measures of the model were STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 51 accuracy = 0.91, AUC = 0.69, TPR = 0.42, FPR = 0.97, MCC = 0.45 and F score was 0.49. Additional model performance measures are in Table 17. Ridge (accuracy). A ridge regression was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: hospitalization (100), PRMS (53.3), RRMS (41.2), center 2 (37.7), other phenotype (36.1), thoracic MRI (33.5), relapse (26.7), center 3 (22.9), SPMS (21.5), center 4 (20.2), cervical MRI (17.4), DMT (2.5), brain MRI (0.67), age (0.49) and gender (0). The confusion matrix predicted 662 true negatives, 59 false positives, 20 false negatives and 18 true positives. Some key measures of the model were accuracy = 0.90, AUC = 0.75, TPR = 0.23, FPR = 0.97, MCC = 0.28 and F score was 0.31. Additional model performance measures are in Table 18. Random forest (accuracy). A random forest was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: hospitalization (100), RRMS (46.0), center 2 (43.4), center 3 (34.0), thoracic MRI (32.6), age (32.1), cervical MRI (28.9), DMT (28.3), center 4 (27.4), relapse (23.7), gender (22.9), other phenotype (21.6), brain MRI (20.3), SPMS (18.8), and PRMS (0). The confusion matrix predicted 665 true negatives, 50 false positives, 17 false negatives and 27 true positives. Some key measures of the model were accuracy = 0.91, AUC = 0.75, TPR = 0.35, FPR = .98, MCC = 0.42 and F1 score was 0.45. Additional model performance measures are in Table 19. All-UC models (accuracy). Evaluating models for AUC, logistic regression performed the best at (0.76), random forest (0.75), ridge (0.75), and LASSO (0.69) (See Figure 11). Logistic regressions AUC was 1.3% higher than random forest, which was the best performing machine learning model. The MCC for LASSO was (0.45), random forest (0.42) logistic regression (0.41), and ridge (0.28). The F1 score for LASSO was (0.49), logistic regression (0.46), random STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 52 forest was (0.44), and ridge (0.31). LASSO outperformed logistic regression on MCC and F1 score by 10% and 6.5% respectively. Figure 12 shows the gain curves across models and upon visual inspection all models need to test about 12.5% of the sample population to capture 50% of positive responses, with the exception of LASSO that requires around 22%. Additional model performance measures are in Table 23. Ridge (MCC). A ridge regression was conducted controlling for confounding effects of age, gender, and center. The models variable importance was as follows: hospitalization (100.0), PRMS (50.8), other phenotype (39.9), RRMS (35.0), SPMS (29.9), center 2 (28.6), relapse (25.7), thoracic MRI (23.6) center 4 (18.7), center 3 (15.3), cervical MRI (10.3), brain MRI (3.4), gender (3.3), DMT (2.5), and age (0). The confusion matrix predicted 648 true negatives, 46 false positives, 34 false negatives and 31 true positives. Some key measures of the model were accuracy = 0.90, AUC = 0.76, TPR = 0.40, FPR = 0.95, MCC = 0.38 and F1 score was 0.44. Additional model performance measures are in Table 20. Random Forest (MCC). A random forest was conducted controlling for confounding effects of age, gender and center. The models variable importance was as follows: hospitalization (100), RRMS (51.0), center 2 (44.1), thoracic MRI (34.9), center 3 (33.8), DMT (30.0), cervical MRI (30.0), age (29.8), center 4 (26.5), gender (23.2), relapse (22.9), other phenotype (21.9), SPMS (21.7), brain MRI (21.5), and PRMS (0). The confusion matrix predicted 668 true negatives, 50 false positives, 14 false negatives and 27 true positives. Some key measures of the model were accuracy = 0.92, AUC = 0.77, TPR = 0.35, FPR = 0.98, MCC = 0.44 and F1 score was 0.46. Additional model performance measures are in Table 21. All-UC models (MCC). For the MCC optimization, logistic regression and LASSO results were the same as accuracy. Evaluating models for AUC, all models performed similarly STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 53 with the exception of LASSO; random forest (0.76), ridge (0.76), logistic regression (0.76), and LASSO (0.69) (See Figure 13). The MCC for LASSO (0.45), random forest (0.44), logistic regression (0.41), and ridge (0.38). The F1 score for LASSO (0.49), logistic regression (0.47), random forest (0.46), and ridge (0.44). The LASSO outperformed logistic regression on MCC and F1 score by 10% and 4% respectively. Logistic regression outperformed ridge for MCC and random forest and ridge for F1 score. Figure 14 shows the gain curves across models and upon visual inspection all models need to test about 12.5% of the sample population to capture 50% of positive responses, with the exception of LASSO that requires around 22%. Additional model performance measures are in Table 23. All-UC indices. The delta between AUC, MCC, and F1 score was added and compared for each model pair. When comparing all-cause UC indices across models logistic regression performed similarly to random forest and LASSO regardless of internal cross-validation optimization. Ridge performed worse overall compared to logistic regression. For perfA there was a -1.1% difference between random forest and logistic regression for accuracy optimization and perfM a 1.58% difference for MCC optimization. Random forest and logistic performed similarly on all their indices. The perfA1 index showed that LASSO and logistic regression performed similar (1.631 vs. 1.639), or a -0.5% difference. For perfA2 ridge performed significantly worse over logistic regression (-17.8%). The perfM2 (MCC optimization) ridge performed worse than logistic regression (-3.8%). See table 25 and figures 16 and 17 for comprehensive data tabulations and visualizations. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 54 Chapter 4 Discussion and Conclusions MS has been studied through basic and clinical trials research and epidemiological studies on population outcomes and quality of life. The MS-CQI adds system-level approaches to study system level effects on population health outcomes using an LHS approach. This study expands upon MS-CQI by moving towards predictive analytics. There is various research using traditional statistical methods, but a lack of research on predictive analytics for MS outcomes. There is a dearth of research on predicting relapse in MS quality improvement collaboratives using predictive analytics, as well as predicting MS relapse and all-cause UC. Relapse and UC utilization are aspects of MS that are costly and disabling, and to date, have been largely unpredictable. Predictive analytics methods have the potential to make these outcomes more predictable than standard approaches and could help shift MS care from a reactive to a predictive approach. The main purpose of this research is comparing statistical model classes between traditional methods and more sophisticated predictive analytics approaches. There is some literature suggesting what are the best model comparison approaches. Zhang et al. (2019) conducted research on a model comparison between random forest and logistic regression in predicting diabetes, and calculated AUC, sensitivity and accuracy for model comparison (2019). Couronn et al. (2018), conducted research and used AIC, AUC, and Brier score and showed the delta between logistic regression and random forest. Couronn et al.s (2018) study is limited as they only compared a few different classification measures and excluded measures such as F1 Score, and Matthews correlation coefficient (MCC) which this study will build upon. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 55 In utilizing predictive analytics for MS relapse literature showed some traditional statistical modeling approaches. Research has been used to determine predictors of MS relapse utilizing Poisson regression (Held et al., 2005), predicting RRMS utilizing discrete distribution models to characterize relapsing-remitting CEL (Velez de Mendizabal et al., 2013) and utilizing a multivariate Cox analysis, to predict future relapses in the short term (Sormani et al., 2007). While these statistical approaches are appropriate, other sophisticated predictive modeling approaches could also be evaluated. For Aim 1 of this study, the predictive modeling findings on MS relapse has filled an important gap in literature to compare traditional statistical models to predictive analytics. In addition, the methods used to evaluate the fit and performance of the models have been broadened by creating a more comprehensive index than what is currently in literature that do not include the use of MCC or F1 score (Couronn et al., 2018). In regards to predicting all-cause UC there is some research on predicting MS hospital utilization. A recent study forecasted MS hospitalization using an Autoregressive Integrated Moving Average (ARIMA; Sharma et al., 2019). There are cost of MS and cost-effectiveness studies estimating current costs and quality of life of patients treated with DMDs (Kobelt et al., 2006). These studies are compelling; however, they do not focus on the endpoint of predicting all-cause UC and do not use predictive analytics and critically evaluate their performance. For Aim 2, this study has filled a significant gap in literature not only with its endpoint of all-cause UC but to compare traditional statistical models to predictive analytics. In addition, the methods used to evaluate the fit and performance of the models are more comprehensive. Reviewing these numerous models through a data science lens via cross- validation and picking a champion model has made the model more generalizable. Comparing different modeling paradigms to establish optimal predictive approaches for relapses and all-cause UC STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 56 could be a trailblazer for the MS community if it is powerful enough to help MS care systems predict, stratify, and address risk in MS populations. All-cause UC has not been researched in depth, various models have not been compared for performance and fit. There are no current uses of modern predictive analytics techniques such as random forest, ridge or LASSO regression for these endpoints in the MS world. This study introduces predictive analytics to the MS field. Quantitative Discussion Aim 1: Relapse Predictive Model Comparison Results substantiate that in certain circumstances machine-learning algorithms outperform traditional statistical models. This supports recent popularity of utilizing machine learning algorithm-based approaches such as random forest. Evaluating relapse indices across models, random forest significantly outperformed logistic regression and other machine learning algorithms regardless of internal cross-validation optimization. For the indices calculated in this study, the delta between AUC, MCC, and F1 scores were added and compared for each model pair to create an index. For perfA (perfA=perfRF perfLR)there was a 0.23 or 27.1% difference between random forest and logistic regression for accuracy optimization. Decoupling the index, the optimal AUC was 0.69 by the random forest accuracy tuned model, which was 0.06 or 9.5% better than logistic regression (0.63). Couronn et als (2018) study showed a mean average difference of 0.04 between RF and LR. This study shows an above average difference on that benchmark measure. MCC for random forest was 0.17 versus 0.08 for logistic, a 0.08 or 100% difference. The F1 Score was 0.20 for random forest versus 0.12 for logistic regression, a 0.08 or 69% difference. Random forest outperformed logistic regression for perfA. For perfM (perfM=perfRFperfLR) there was a 0.23 or 27.5% difference between random forest and logistic regression. The AUC on RF was 9.5% higher LR, same as perfA. In addition, STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 57 random forest performed better on MCC and F1 score. These are significant differences when looking at the overall composite index or individual measures. As discussed in the literature MCC is a more appropriate way to evaluate imbalanced datasets (Chicco, & Jurman, 2020). The MMC is not affected by the unbalanced datasets, and is a contingency matrix method of calculating the Pearson product- moment correlation coefficient between actual and predicted values (Chicco, & Jurman, 2020). Thus, the MCC is a reliable statistical measure and produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (Chicco & Jurman, 2020). For the prediction for perfM,, the true positives were a bit higher in logistic regression (11 vs. 9) compared to random forest. Logistic regression had almost three times the number of false negatives compared to random forest (148 vs. 53) as well as logistic predicting significantly less true negatives (582 vs. 677). The difference between predictive false positives were minimum between RF and LR (20 vs. 18). The number of false negatives in each model drives the overall performance of the logistic model and its associated utility down. Random forest outperformed logistic regression for perfM. The imbalanced nature of this dataset limits overall predictive capability, however, RF does a significantly better job at overall prediction across most measures. The increased performance and fit of RF is substantial and does a much better job of predicting true negatives for MS relapse compared to LR. The number of false negatives that LR produces substantiates the use of RF over other modeling techniques when predicting MS relapse. One of the limitations of the dataset was the imbalance. Relapse is low prevalence and the amount of non-events within the data makes it difficult to make a prediction as some of the statistical models will predict all non-events for relapse. However, as the results have shown, STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 58 being able to fine tune models within data science paradigm gives the outcome more flexibility to be accurate. The ability to use different internal cross-validation optimization techniques and oversampling in the data science space allowed further agility and flexibility during the modeling process. In addition, having a wide array of modeling choices beyond traditional statistical modeling allowed greater comparison of different modeling techniques and the opportunity for the best model. While the prevalence is low, adding more data may increase the power and robustness of the models. Quantitative Discussion Aim 2: All-Cause UC Predictive Model Comparison Results here suggest that machine learning algorithms can perform similarly to traditional statistical models. Here, logistic regression significantly outperformed ridge regression. This finding is an opportunity to better understand the conditions when machine learning algorithms are not preferable to standard statistical methods. When evaluating all-cause UC indices across models logistic regression performed similarly to random forest and LASSO regardless of internal cross-validation optimization. For perfA there was a -0.02 or -1.1% difference between random forest and logistic regression for accuracy optimization and perfM a 0.03 or 1.58% difference. Decoupling the perfM index, the optimal AUC was 0.77 by the random forest versus 0.76 for logistic regression a nominal difference. This does not support findings from Couronn et al. (2018) which demonstrated a mean average difference of 4.1% between RF and LR for AUC. This is a below average difference on that benchmark measure. However, MCC for random forest was 0.44 versus 0.41 for logistic, a 0.3 or 7.3% difference. For F1 Score there was minimal difference between RF and LR (.458 vs. .466). For perfA random forest vs logistic regression on AUC was similar (0.75 vs. 0.76). In addition, both models performed similar on both MCC and F1 Score. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 59 These are not significant differences when looking at the overall composite index or individual measures. Random forest did not outperform logistic regression on any of the models on this particular measure as well as models optimized for accuracy. There was a negligible difference between the performance of random forest and logistic regression models for perfA. and perfM. For perfA1 (perfA1=perfLASSOperfLR) comparing LASSO to LR there is a 0.01 (-0.5%) difference on the index. The LR model outperformed LASSO on AUC (0.76 vs. 0.69). However, lasso performed better on the MCC metric (0.45 vs. 0.41). Evaluating LR and LASSO overall via the index there is not much difference between the two models. For perfA2 (perfA2=perfRRperfLR) there was a -0.29 or -17.8% difference between the two model indices. The AUC for ridge was 0.751 versus 0.763 for LR, which is not a significant difference. However, when reviewing MCC and F1 score the differences were substantial (0.28 vs. 0.41 and 0.31 vs. 0.46) respectively. This reinforces why an index that includes multiple measures is so pertinent for model fit and performance. Ridge regression had more false positives and less true positives in the prediction (l59 vs. 42 and 18 vs. 34). Logistic regression significantly outperformed ridge regression for perfA2. The all-cause UC predictive models demonstrate fairly good fit and performance in both random forest and logistic regression. There is not a significant difference between ML algorithms and traditional logistic regression, however, logistic regression does outperform ridge regression. There is an opportunity to better understand in which circumstances ML algorithms may perform better with certain datasets and outcomes such as MS relapse. Limitations This study is a secondary analysis based on a Step-Wedge RCT. Randomized clinical trials are the gold standard in research design for inferring a causal relationship (Polit & Beck, STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 60 2009). However, there are threats to external validity based on the generalizability of the results. External validity is the extent to which the results of an RCT can be generalized to the general population and in clinical practice (Rothwell, 2005). This analysis was based on year 3 of the study, which is a cross sectional design and is limited to finding associations, and cannot determine causation. The final results of the three-year study which is longitudinal, may be able to thoroughly investigate and confirm the findings presented in this research. The population observed in the parent MS-CQI study is from a sample of MS care centers in the eastern U.S, and therefor results may have generalizability limited to that geographic region. This geographic limitation impacts the generalizability because it only includes four MS centers located in the Eastern U.S. However, the sample also consists of general demographic characteristics that are representative of PwMS such as majority female and RRMS, which is a strength of the data. An additional limitation was the inclusion of a limited number of variables from the electronic medical records in the parent MS-CQI study. The inclusion of additional variables that are available within an EHR, such as diagnosis codes and lab results that may have added additional features to the model to increase its predictive power and overall performance. With both study aims, additional machine learning models could be added and compared such as K Nearest Neighbors, Nave Bayes and Support Vector Machine. Additional R packages may be available to those with advanced data science expertise that are not utilized as part of this study. Additional data may be advantageous to promote a more robust model for both research aims. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 61 Implications on Future Research There is an opportunity for future research regarding Aim 1 of this study. Predicting relapse is crucial in MS care. As OConnell et al. (2014) notated, relapses can result in incomplete recovery and have a significant impact on PwMS quality of life. The ability to apply machine-learning methods such as random forest to predict relapse more accurately could improve the overall quality of life of people with MS as well as save on medical cost. Paired with a learning health system such as MS-CQI which is configured to utilize feed forward predictive analytic data to inform intelligent action, powerful predictive analytics could help convert MS care practice from a reactive to a predictive stance. For example, predictive analytics could help MS care centers identify subpopulations at elevated risk of relapse and take preventive action before the relapses actually occur. In MS-CQI, the majority of patients had RRMS; therefore, there is opportunity to conduct a subsequent independent study of this subpopulation to focused on optimizing relapse outcomes. There is not much of a cost difference in implementing predictive analytics on smaller scale research. Data science software such as R and Python are open source and are free. If the dataset is small a standard laptop can perform the data wrangling and modeling the cost should be nominal. For smaller scale projects this may be a cost savings as opposed to statistical software that has a license fee associated to use the product. Costs to deploy machine learning on a large scale should be compared to traditional statistical methods to determine utility, cost, and performance. Data scientists are in high demand, costs may be higher for a data scientist versus a traditional statistician. These factors should be evaluated as part of your analytics plan when embarking on new research. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 62 All-cause UC is important in the MS community. The ability to introduce predictive analytics into MS could change health care delivery for MS. Although there was not a big difference in various predictive models, there is a dearth of research on this research aim. This research could improve the overall quality of life of people with MS as well as save on medical cost. In addition, the majority of these patients had RRMS, there is opportunity to conduct a subsequent independent study of this subpopulation to focus on identifying high risk subpopulations and using LHS-oriented improvement and population health approaches to reduce relapse rate. Conclusion In this study, logistic regression has been compared to random forest, ridge regression, and LASSO for the first time for feature selection and classification of MS relapse and MS allcause UC utilization in a population of people with MS followed by four MS centers participating in a LHS model improvement science research collaborative. This study has filled important gaps in literature using a MSCQI based framework. Comparing the predictability of relapse across various models with a predictive analytics framework can potentially change how we manage MS care. Imagine that people with MS could get care and support before a relapse happens because of predictive analytics identifying the patient as high risk, rather than react to it after it happens. This would potentially reduce the risk of disability progression further than could be possible in conventional care. There are healthcare quality outcomes and cost implications beyond the methodological comparisons of statistical models as theoretical research. Predictive analytics methods have the potential to empower and accelerate the capability of LHS to predict and respond to emerging health needs of people with costly, complex, chronic conditions like MS and better inform STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 63 continuous improvement efforts. Predictive analytics could make LHS cheaper, faster, better and could inform research and improvement in new ways and accelerate improvement science research to make MS care better and ultimately, outcomes better for PwMS. This would potentially result in achieving more efficient healthcare outcomes and saving time and overall cost. Sophisticated predictive modeling can make an unpredictable disease like MS more predictable in various outcomes such as relapses and all-cause UC. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 64 References Backman, D. R., Kohatsu, N. D., Yu, Z., Abbott, R. E., & Kizer, K. W. (2017). Peer Reviewed: Implementing a Quality Improvement Collaborative to Improve Hypertension Control and Advance Million Hearts Among Low-Income Californians, 20142015. Preventing Chronic Disease, 14. Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A., & Nielsen, H. (2000). Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics, 16(5), 412-424. Benneyan, J. C., Lloyd, R. C., & Plsek, P. E. (2003). Statistical process control as a tool for research and healthcare improvement. BMJ Quality & Safety, 12(6), 458-464. http://dx.doi.org/10.1136/qhc.12.6.458 Bozkaya, D., Livingston, T., Migliaccio-Walle, K., & Odom, T. (2017). The cost-effectiveness of disease-modifying therapies for the treatment of relapsing-remitting multiple sclerosis. Journal of Medical Economics, 20(3), 297-302. doi:10.1080/13696998.2016.1258366 Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199-231. doi:10.1214/ss/1009213726 Cartwright, N. (2010). What are randomised controlled trials good for?. Philosophical studies, 147(1), 59. https://doi.org/10.1007/s11098-009-9450-2 Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321357. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 65 Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(6), 113. https://doi.org/10.1186/s12864-019-6413-7 Clinical microsystems: Where quality, safety and value are made (n.d.). Retrieved from http://www.clinicalmicrosystem.org/ Fanjiang, G., Grossman, J. H., Compton, W. D., & Reid, P. P. (Eds.). (2005). Building a better delivery system: a new engineering/health care partnership. National Academies Press. https://doi.org/10.17226/11378 Couronn, R., Probst, P., & Boulesteix, A. L. (2018). Random forest versus logistic regression: a large-scale benchmark experiment. BMC Bioinformatics, 19(1), 270. Crohns & Colitis Foundation. (n.d.). Quality of Care: IBD Qorus. Retrieved from https://www.crohnscolitisfoundation.org/research/ibd-qorus Definition of MS: National Multiple Sclerosis Society (2019). Retrieved from https://www.nationalmssociety.org/What-is-MS/Definition-of-MS de Silva, D. (2014). Improvement collaboratives in health care. (Report No. 21). London: Health Foundation [PDF file]. Retrieved from https://www.health.org.uk/sites/default/files/ImprovementCollaborativesInHealthcare.pdf Gisler, S. (2019, May 21). EMD Serono Launches MS-LINK Research Network to improve patient care. Retrieved from https://multiplesclerosisnewstoday.com/newsposts/2019/05/07/emd-serono-launches-ms-link-a-community-to-advance-ms-scienceand-improve-patient-lives/ STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 66 Godfrey, M. M., & Oliver, B. J. (2014). Accelerating the rate of improvement in cystic fibrosis care: Contributions and insights of the learning and leadership collaborative. BMJ Quality & Safety, 23(Suppl 1), i23-i32. doi:10.1136/bmjqs-2014-002804 Gustafson, D. H., Quanbeck, A. R., Robinson, J. M., Ford, J. H., Pulvermacher, A., French, M. T.,...McCarty, D. (2013). Which elements of improvement collaboratives are most effective? A clusterrandomized trial. Addiction, 108(6), 1145-1157. doi:10.1111/add.12117 Hajian-Tilaki, K. (2013). Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian Journal of Internal Medicine, 4(2), 627-635. Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377-381. https://doi.org/10.1016/j.jbi.2008.08.010 Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L.,...Duda, S. N. (2019). The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics, 95, 103208. https://doi.org/10.1016/j.jbi.2019.103208 Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The elements of statistical learning: Data mining, inference, and prediction. (2nd ed.). New York: Springer. Held, U., Heigenhauser, L., Shang, C., Kappos, L., & Polman, C. (2005). Predictors of relapse rate in MS clinical trials. Neurology, 65(11), 1769-773. doi:10.1212/01.wnl.0000187122.71735.1f STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 67 Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, 1-7. doi:10.1136/bmj.h391 Hulscher. M., Schouten, L., & Grol, R. (2009). Collaboratives. London: Health Foundation [PDF file]. Retrieved from https://www.health.org.uk/sites/default/files/Collaboratives.pdf Iezzoni, L. I., & Ngo, L. (2007). Health, disability, and life insurance experiences of workingage persons with multiple sclerosis. Multiple Sclerosis Journal, 13(4), 534-546 Institute for healthcare improvement: The IHI triple aim. (n.d.). Retrieved from from http://www.ihi.org:80/Engage/Initiatives/TripleAim/Pages/default.aspx Institute for Healthcare Improvement (2014). Plan-Do-Study-Act (PDSA) worksheet. Retrieved from http://www.ihi.org/knowledge/Pages/Tools/PlanDoStudyActWorksheet.aspx Institute for Healthcare Improvement (2003). The Breakthrough Series: IHIs Collaborative Model for Achieving Breakthrough Improvement [PDF file]. Retrieved from http://www.ihi.org/resources/Pages/IHIWhitePapers/TheBreakthroughSeriesIHIsCollabor ativeModelforAchievingBreakthroughImprovement.aspx Institute of Medicine (2001). Crossing the quality chasm: A new health system for the 21st century [PDF file]. Retrieved from http://www.iom.edu/~/media/Files/Report%20Files/2001/Crossing-the-QualityChasm/Quality%20Chasm%202001%20%20report%20brief.pdf James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning: With applications in R. New York, NY: Springer. Jones, E., Pike, J., Marshall, T., & Ye, X. (2016). Quantifying the relationship between increased disability and health care resource utilization, quality of life, work productivity, health STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 68 care costs in patients with multiple sclerosis in the US. BMC Health Services Research, 16(1), 294-302. doi:10.1186/s12913-016-1532-1 Kilo, C. M. (1998). A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement's Breakthrough Series. Quality Management in Health Care, 6, 1-14. Kirasich, K., Smith, T., & Sadler, B. (2018). Random forest vs logistic regression: Binary classification for heterogeneous datasets. SMU Data Science Review, 1(3), 1-24. Retrieved from https://scholar.smu.edu/datasciencereview/vol1/iss3/9 Kobelt, G., Berg, J., Atherly, D., & Hadjimichael, O. (2006). Costs and quality of life in multiple sclerosis. Neurology, 66(11), 1696-702. doi:10.1212/01.wnl.0000218309.01322.5c Koch-Henriksen, N., Thygesen, L. C., Srensen, P. S., & Magyari, M. (2019). Worsening of disability caused by relapses in multiple sclerosis: A different approach. Multiple sclerosis and related disorders, 32, 1-8. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development (Vol. 1). Englewood Cliffs, NJ: Prentice-Hall. Lean Six Sigma Online. (n.d.). What are the Lean Six Sigma principles? Retrieved from https://www.purdue.edu/leansixsigmaonline/blog/lean-six-sigma-principles/ Lindblad, S., Ernestam, S., Van Citters, A. D., Lind, C., Morgan, T. S., & Nelson, E. C. (2017). Creating a culture of health: Evolving healthcare systems and patient engagement. QJM: An International Journal of Medicine, 110(3), 125-129. https://doi.org/10.1093/qjmed/hcw188 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 69 Matthews, B. W. (1975). Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2), 442451. https://doi.org/10.1016/0005-2795(75)90109-9 Meninno, E., Livingston, T., Stuart, S., Powell, S., Ahmad, A., Afsari, N., ... & Tornatore, C. (2018). Observations of a multiple sclerosis patient-centered specialty practice: Analysis of depression patterns. Neurology Apr 2018, 90 (15 Supplement) P4. 425. Retrieved from https://n.neurology.org/content/90/15_Supplement/P4.425.abstract Multiple Sclerosis Partners Advancing Technology and Health Solutions. (2016). Seeking to enhance MS care. Retrieved from https://www.mspaths.com Nelson, E. C., Greenfield, S., Hays, R. D., Larson, C., Leopold, B., & Batalden, P. B. (1995). Comparing outcomes and charges for patients with acute myocardial infarction in three community hospitals: An approach for assessing "value". International Journal for Quality in Health Care, 7(2), 95-108. doi:10.1093/intqhc/7.2.95 Nelson, E. C., Batalden, P. B., Godfrey, M. M., & Lazar, J. S. (2011). Value by design: developing clinical microsystems to achieve organizational excellence. John Wiley & Sons Nelson, E. C., Dixon-Woods, M., Batalden, P. B., Homa, K., Van Citters, A. D., Morgan, T. S.,...Lindblad, S. (2016). Patient focused registries can improve health, care, and science. BMJ, 354, 1-6. doi:10.1136/bmj.i3319 North American Research Committee on Multiple Sclerosis Sclerosis, (2017). NARCOMS registry for multiple sclerosis. Retrieved from https://www.narcoms.org/ STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 70 O'Connell, K., Kelly, S. B., Fogarty, E., Duggan, M., Buckley, L., Hutchinson, M.,...Tubridy, N. (2014). Economic costs associated with an MS relapse. Multiple Sclerosis and Related Disorders, 3(6), 678-683. doi:10.1016/j.msard.2014.09.002 Oliver, B. J. (2019, October, 2019). The Multiple Sclerosis Continuous Improvement Collaborative: A coproduction research learning health system for costly complex chronic illness care. Platform presentation at The International Society for Quality in Health Care (ISQua) Annual International Conference. Cape Town, South Africa Oliver, B. J., Messier, R., Hall, A. for the MSCQI Investigators, (2019, June 2019). The Multiple Sclerosis Continuous Improvement Collaborative: Improvement science and coaching applications in a multicenter outcomes research study. Paper presented at the Sheffield Microsystem Coaching Academy Annual International Expo. Sheffield, United Kingdom Polit, D. F., & Beck, C. T. (2009). International differences in nursing research, 2005 2006. Journal of Nursing Scholarship, 41(1), 44-53. Porter, M. E. (2009). A strategy for health care reformtoward a value-based system. New England Journal of Medicine, 361(2), 109-112. Quality of Care: IBD Qorus (n.d.). Retrieved from https://www.crohnscolitisfoundation.org/research/ibd-qorus Quinn, J. B. (1992). Intelligent enterprise: A knowledge and service based paradigm for industry. Free Press, New York, USA. R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from http://www.R-project.org/ STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 71 Riccio, P., Rossano, R., Larocca, M., Trotta, V., Mennella, I., Vitaglione, P., ...Coniglio, M. G. (2016). Anti-inflammatory nutritional intervention in patients with relapsing-remitting and primary-progressive multiple sclerosis: A pilot study. Experimental Biology and Medicine, 241(6), 620-635. https://doi.org/10.1177/1535370215618462 Sharma, K., Bittner, F., & Cho, T. (2019). Forecasting multiple sclerosis hospitalization and healthcare burden in the United States from 2017 to 2040. Neurology, 92(Suppl 15), P4. 6-007. imundi A. M. (2009). Measures of diagnostic accuracy: Basic definitions. eJournal of the International Federation of Clinical Chemistry and Laboratory Medicine, 19(4), 203 211. Sormani, M. P., Rovaris, M., Comi, G., & Filippi, M. (2007). A composite score to predict shortterm disease activity in patients with relapsing-remitting MS. Neurology, 69(12), 12301235. doi:10.1212/01.wnl.0000276940.90309.15 Stoppe, M., Busch, M., Krizek, L., & Then Bergh, F. (2017). Outcome of MS relapses in the era of disease-modifying therapy. BMC Neurology, 17(1), 151-159. doi:10.1186/s12883-0170927-x Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2019). Using multivariate statistics (7th ed.). Boston, MA: Pearson. Thor, J., Lundberg, J., Ask, J., Olsson, J., Carli, C., Hrenstam, K. P., & Brommels, M. (2007). Application of statistical process control in healthcare improvement: Systematic review. BMJ Quality & Safety, 16(5), 387-399. doi:10.1136/qshc.2006.022194 Types of MS: National Multiple Sclerosis Society (2019). Retrieved from https://www.nationalmssociety.org/What-is-MS/Types-of-MS STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 72 Velez de Mendizabal, N., Hutmacher, M. M., Troconiz, I. F., Goni, J., Villoslada, P., Bagnato, F., & Bies, R. R. (2013). Predicting relapsing-remitting dynamics in multiple sclerosis using discrete distribution models: A population approach. PloS One, 8(9), 1-11. doi:10.1371/journal.pone.0073361 Vermont Oxford Network: Who We are. (n.d.). Retrieved from https://public.vtoxford.org/whowe-are-overview/ Vollmer, T. (2007). The natural history of relapses in multiple sclerosis. Journal of the Neurological Sciences, 256, S5-S13. https://doi.org/10.1016/j.jns.2007.01.065 Wallin, M. T., Culpepper, W. J., Campbell, J. D., Nelson, L. M., Langer-Gould, A., Marrie, R. A.,...LaRocca, N. G. (2019). The prevalence of MS in the United States: A populationbased estimate using health claims data. Neurology, 92(10), e1029-e1040. doi:10.1212/wnl.0000000000007035 West, S. G., Duan, N., Pequegnat, W., Gaist, P., Des Jarlais, D. C., Holtgrave, D.,...Mullen, P. D. (2008). Alternatives to the randomized controlled trial. American Journal of Public Health, 98(8), 1359-1366. doi:10.2105/AJPH.2007.124446 What are the Lean Six Sigma Principles? (n.d.). Retrieved from https://www.purdue.edu/leansixsigmaonline/blog/lean-six-sigma-principles/ Zhang, B., Lu, L., & Hou, J. (2019). A comparison of logistic regression, random forest models in predicting the risk of diabetes. ISICDM 2019: Proceedings of the Third International Symposium on Image Computing and Digital Medicine. 231-234. https://doi.org/10.1145/3364836.3364882. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 73 Table 1 Patient Characteristics for Year 3 by Center Center 1 Characteristics Center 2 Center 3 Center 4 Total Number Percent Number Percent Number Percent Number Percent Number Percent 977 38.6 460 18.2 539 21.3 556 22.0 2,532 100 RRMS 787 80.6 364 79.1 393 72.9 518 93.2 2,062 81.4 SPMS 89 9.1 43 9.3 78 14.5 18 3.2 228 9.0 PPMS 59 6.0 28 6.1 37 6.9 12 2.2 136 5.4 PRMS 0 0.0 0 0.0 0 0.0 7 1.3 7 0.3 Other 42 4.3 25 5.4 31 5.8 1 0.18 99 3.9 51.3 12.2 48.3 13.2 50.6 12.8 48.0 12.9 49.9 12.7 Female 764 78.2 332 72.2 384 71.2 437 78.6 1,917 75.7 Male 213 22.0 128 27.9 155 28.8 119 21.4 615 24.3 Participants Phenotype*** Age (mean, SD) *** Sex*** Note. Chi square for categorical variables, ANOVA for continuous variables. * p<.05, ** p<.01, *** p<.001. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 74 Table 2 Outcomes for Year 3 by Center Center 1 Outcome Center 2 Center 3 Center 4 Total Number Percent Number Percent Number Percent Number Percent Number Percent Participants 977 38.6 460 18.2 539 21.3 556 22.0 2,532 100 >=1DMT*** 719 73.6 274 59.6 327 60.7 526 94.6 1,846 72.9 Brain MRI 498 51.0 209 45.4 199 36.9 311 55.9 1,217 48.1 Cervical MRI*** 338 34.6 112 24.4 99 18.4 129 23.2 678 26.8 Thoracic MRI*** 142 14.5 106 23.0 21 3.9 101 18.2 370 14.6 Hospitalizations*** 57 5.8 49 10.7 23 4.3 47 8.5 176 7.0 ER Visits*** 79 8.1 51 11.1 31 5.8 46 8.3 207 8.2 UC Visits*** 34 3.5 24 5.2 9 1.7 1 0.2 68 2.7 Relapses*** 33 3.4 14 3.0 14 2.6 36 6.5 97 3.8 0.07 0.31 0.15 0.48 0.05 0.23 0.09 0.32 0.09 0.34 0.10 0.36 0.15 0.45 0.06 0.28 0.09 0.30 0.1 0.35 0.04 0.21 0.06 0.27 0.02 0.13 0.002 0.04 0.03 0.19 0.04 0.20 0.04 0.23 0.03 0.18 0.06 0.27 0.04 0.22 Hospitalization (mean, SD)** ED Utilization (mean, SD)** UC Utilization (mean, SD)** Relapse Rate (mean, SD)** Note. Chi square for categorical variables, ANOVA for continuous variables. * p<.05, ** p<.01, *** p<.001. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 75 Table 3 Patient Characteristics and Outcomes by Derivation and Validation Cohort Total Characteristic Derivation Cohort Validation Cohort Number Percent Number Percent Number Percent Participants 2,532 100 1,773 70.0 759 30.0 Age (mean, SD) 49.9 12.7 49.9 12.8 49.9 12.6 615 24.2 434 24.4 181 13.5 1,917 75.8 1,339 75.6 578 86.5 No 470 18.5 313 17.6 157 20.1 Yes 2,062 81.5 1,460 82.4 602 79.9 No 2,304 90.9 1,619 92.3 685 90.2 Yes 228 9.1 154 7.7 74 9.8 No 2,396 94.6 1,684 95.0 712 93.8 Yes 136 5.4 89 5.0 47 6.2 Sex Male Female Phenotype RRMS SPMS PPMS STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Total Characteristic 76 Derivation Cohort Validation Cohort Number Percent Number Percent Number Percent No 2,525 99.7 1,770 99.8 755 99.5 Yes 7 0.0 3 0.0 4 0.0 No 2,433 96.0 1,706 96.2 727 95.8 Yes 99 4.0 67 3.8 32 4.2 No 686 27.1 468 26.4 218 28.7 Yes 1,846 72.9 1,305 73.6 541 71.3 No 1,315 51.9 901 50.8 414 54.5 Yes 1,217 48.1 872 49.2 345 45.5 No 1,854 73.2 1,283 72.3 571 75.2 Yes 678 26.8 490 27.7 188 24.3 2,274 89.8 1,592 89.8 682 89.8 PRMS Other DMT Brain MRI Cervical MRI All UC No STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Total Characteristic 77 Derivation Cohort Validation Cohort Number Percent Number Percent Number Percent 258 10.2 181 10.2 77 10.2 No 2,162 85.4 1,506 86.4 656 86.4 Yes 370 14.6 267 13.6 103 13.6 No 2,356 93.0 1,651 93.1 705 92.9 Yes 176 7.0 122 6.9 54 7.1 No 2,325 91.8 1,628 91.8 697 91.8 Yes 207 8.2 145 8.2 62 8.2 No 2,464 97.3 1,725 97.3 739 97.4 Yes 68 2.7 48 2.7 20 2.6 No 2,435 96.2 1,699 95.8 736 96.9 Yes 97 3.8 74 4.2 23 3.1 Yes Thoracic MRI Hospitalizations ED Visits UC Visits Relapse STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 78 Table 4 Bivariate Analysis (n = 2,532) Variable 1 2 3 4 5 6 7 8 9 10 11 12 1. Phenotype 1 30.0*** 164.2*** 30.3*** 11.3* 10.5* 16.6* 4.1 1.06 7.6 122.1*** 2. Age 1 3. Gender 30.0*** 1 1.0 1.32 0.06 0.81 0.1 .40 4.6* 0.38 14.8* 4. DMT 164.2*** 1.0 1 177.2*** 64.8*** 45.4*** 2.1 .10 0.45 8.2** 215*** 5. Brain MRI 30.3*** 1.3 177.2*** 1 781.1*** 329.3*** 2.2 5.7* 0.67 29.9*** 45*** 6. Cervical MRI 11.3* .06 64.8*** 781.1*** 944.8*** 1.47 9.3** 3.6 31.6*** 54.9*** 7. Thoracic MRI 10.5* .81 45.4*** 329.3*** 944.8*** 1 6.2* 11.8** 6.0* 33.8*** 81.4*** 8. Hospitalization 16.6** .10 2.1 2.2 1.47 6.2* 1 582.3*** 20.1*** 11.3** 19.6*** 9. ED Visit 4.1 .40 0.1 5.7* 9.25** 11.8** 582.3** 1 26.3*** 17.5*** 9.4* 10. UC Visit 1.06 4.6* 0.45 .67 3.55 6.0* 20.1*** 26.3*** 1 0.06 29.1*** 11. Relapse 7.6 .38 8.2** 29.9*** 31.6*** 33.8*** 11.3** 17.5*** 0.06 1 14.1** 12. Center 122*** 14.8** 215.1*** 45.2*** 54.9*** 81.4*** 19.6** 9.4* 29.1*** 14.1** 1 1 Note. Chi square for categorical variables, Pearson r for continuous variables. ED=emergency department; UC=urgent care. * p <0.05, ** p<0.01, *** p<0.001. Test could not be performed = . STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 79 Table 5 Model Comparison Metrics Metric Accuracy Formula (true positives + true negatives) / (true positives + true negatives + false positives + false negatives) True Positive Rate (TPR)Sensitivity true positives / (true positives + false negatives) False Positive Rate (FPR)Specificity false positives / (false positives + true negatives) Negative Predictive Value (NPV) True negatives / (true negatives + false negatives) Precision (PPV) Area Under the Curve (AUC) true positives / (true positives + false positives) Graphical Plotting of TPR vs. FPR Matthews Correlation Coefficient (MCC) Note. Adapted from Random Forest vs Logistic Regression: Binary Classification for Heterogeneous Datasets, by K. Kirasich, T. Smith and B. Sadler, 2018, SMU Data Science Review, 1(3), p. 13. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 6 Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Stepwise Logistic Regression Metric Result Accuracy 0.781 True Positive Rate (TPR) 0.379 False Positive Rate (FPR) 0.797 Negative Predictive Value (NPV) 0.970 MCC 0.083 Precision 0.069 F Score 0.117 AUC 0.629 80 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 7 Relapse Model Comparison Metrics-Cross-Validated (Accuracy) LASSO Metric Result Accuracy 0.962 True Positive Rate (TPR) 0.000 False Positive Rate (FPR) 1.000 Negative Predictive Value (NPV) 0.962 MCC Precision F Score AUC Note. Test could not be performed = . 0.510 81 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 8 Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Ridge Metric Result Accuracy 0.962 True Positive Rate (TPR) 0.000 False Positive Rate (FPR) 1.000 Negative Predictive Value (NPV) 0.962 MCC Precision F Score AUC Note. Test could not be performed = . 0.670 82 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 9 Relapse Model Comparison Metrics-Cross-Validated (Accuracy) Random Forest Metric Result Accuracy 0.904 True Positive Rate (TPR) 0.310 False Positive Rate (FPR) 0.927 Negative Predictive Value (NPV) 0.971 MCC 0.166 Precision 0.145 F Score 0.198 AUC 0.690 83 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 10 Relapse Model Comparison Metrics-Cross-Validated (MCC) LASSO Metric Result Accuracy 0.773 True Positive Rate (TPR) 0.414 False Positive Rate (FPR) 0.788 Negative Predictive Value (NPV) 0.971 MCC 0.093 Precision 0.071 F Score 0.122 AUC 0.622 84 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 11 Relapse Model Comparison Metrics-Cross-Validated (MCC) Ridge Metric Result Accuracy 0.810 True Positive Rate (TPR) 0.414 False Positive Rate (FPR) 0.826 Negative Predictive Value (NPV) 0.973 MCC 0.119 Precision 0.086 F Score 0.143 AUC 0.645 85 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 12 Relapse Model Comparison Metrics-Cross-Validated (MCC) Random Forest Metric Result Accuracy 0.895 True Positive Rate (TPR) 0.345 False Positive Rate (FPR) 0.916 Negative Predictive Value (NPV) 0.972 MCC 0.172 Precision 0.141 F Score 0.200 AUC 0.685 86 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 13 Relapse Model Comparison Metrics-Cross-Validated (F1) LASSO Metric Result Accuracy 0.962 True Positive Rate (TPR) 0.000 False Positive Rate (FPR) 0.000 Negative Predictive Value (NPV) 0.962 MCC Precision F Score AUC Note. Test could not be performed = . 0.629 87 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 14 Relapse Model Comparison Metrics-Cross-Validated Metric Result Accuracy 0.877 True Positive Rate (TPR) 0.414 False Positive Rate (FPR) 0.896 Negative Predictive Value (NPV) 0.975 MCC 0.185 Precision 0.136 F Score 0.205 AUC 0.653 88 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 15 Relapse Model Comparison Metrics-Cross-Validated (F1) Random Forest Metric Result Accuracy 0.858 True Positive Rate (TPR) 0.207 False Positive Rate (FPR) 0.884 Negative Predictive Value (NPV) 0.966 MCC 0.053 Precision 0.065 F Score 0.10 AUC 0.667 89 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 16 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Stepwise Logistic Regression Metric Result Accuracy 0.897 True Positive Rate (TPR) 0.442 False Positive Rate (FPR) 0.949 Negative Predictive Value (NPV) 0.938 MCC 0.410 Precision 0.493 F Score 0.466 AUC 0.763 90 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 17 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) LASSO Metric Result Accuracy 0.912 True Positive Rate (TPR) 0.416 False Positive Rate (FPR) 0.968 Negative Predictive Value (NPV) 0.936 MCC 0.450 Precision 0.593 F Score 0.489 AUC 0.692 91 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 18 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Ridge Metric Result Accuracy 0.896 True Positive Rate (TPR) 0.234 False Positive Rate (FPR) 0.971 Negative Predictive Value (NPV) 0.918 MCC 0.283 Precision 0.474 F Score 0.313 AUC 0.751 92 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 19 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (Accuracy) Random Forest Metric Result Accuracy 0.912 True Positive Rate (TPR) 0.351 False Positive Rate (FPR) 0.975 Negative Predictive Value (NPV) 0.930 MCC 0.421 Precision 0.614 F Score 0.446 AUC 0.754 93 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 20 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (MCC) Ridge Metric Result Accuracy 0.895 True Positive Rate (TPR) 0.403 False Positive Rate (FPR) 0.950 Negative Predictive Value (NPV) 0.934 MCC 0.381 Precision 0.477 F Score 0.437 AUC 0.758 94 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 21 All-Cause Urgent Care Model Comparison Metrics-Cross-Validated (MCC) Random Forest Metric Result Accuracy 0.916 True Positive Rate (TPR) 0.351 False Positive Rate (FPR) 0.979 Negative Predictive Value (NPV) 0.930 MCC 0.441 Precision 0.659 F Score 0.458 AUC 0.766 95 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 96 Table 22 Relapse Model Comparison Metrics-All Predictive Models Model Accuracy TPR FPR NPV MCC Precision F1 Score AUC Logistic Regression (Accuracy) 0.781 0.379 0.797 0.97 0.083 0.069 0.117 0.629 LASSO (Accuracy) 0.962 0 1 0.962 0.51 Ridge (Accuracy) 0.962 0 1 0.962 0.67 Random Forest (Accuracy) 0.904 0.310 0.927 0.971 0.166 0.145 0.198 0.69 Logistic Regression (MCC) 0.781* 0.379* 0.797* 0.97* 0.083* 0.069* 0.117* 0.629* LASSO (MCC) 0.773 0.414 0.788 0.971 0.093 0.071 0.122 0.622 Ridge (MCC) 0.810 0.415 0.826 0.973 0.119 0.086 0.143 0.645 Random Forest (MCC) 0.895 0.345 0.916 0.972 0.172 0.141 0.2 0.685 Logistic Regression (F1) 0.781* 0.379* 0.797* 0.97* 0.083* 0.069* 0.117* 0.629* LASSO (F1) 0.962 0 1 0.962 0.585 Ridge (F1) 0.877 0.414 0.896 0.975 0.185 0.136 0.205 0.653 Random Forest (F1) 0.858 0.207 0.884 0.966 0.053 0.065 0.10 0.667 Note. * Logistic regression results are the same across all optimization techniques. Test could not be performed = . STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 97 Table 23 All-Cause Urgent Care Model Comparison Metrics-All Predictive Models Model Accuracy TPR FPR NPV MCC Precision F1 Score AUC Logistic Regression (Accuracy) 0.897 0.442 0.949 0.938 0.410 0.493 0.466 0.763 LASSO (Accuracy) 0.912 0.416 0.968 0.936 0.450 0.593 0.489 0.692 Ridge (Accuracy) 0.896 0.234 0.971 0.918 0.283 0.474 0.313 0.751 Random Forest (Accuracy) 0.912 0.351 0.975 0.930 0.421 0.614 0.446 0.754 Logistic Regression (MCC) 0.897* 0.442* 0.949* 0.938* 0.410* 0.493* 0.466* 0.763* LASSO (MCC) 0.912* 0.416* 0.968* 0.936* 0.450* 0.593* 0.489* 0.692* Ridge (MCC) 0.895 0.403 0.950 0.934 0.381 0.477 0.437 0.758 Random Forest (MCC) 0.916 0.351 0.979 0.930 0.441 0.659 0.458 0.766 Note. * Logistic regression results are the same across all optimization techniques. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 24 Performance Relapse Indices Index Result perfA=perfRF perfLR 1.054-.829=0.225 (27.1%) perfA1=perfLASSOperfLR - (MCC and F1 not calculated for LASSO) perfA2=perfRRperfLR - (MCC and F1 not calculated for Ridge) perfM=perfRFperfLR 1.057-.829*=0.228 (27.5%) perfM1=perfLASSOperfLR 0.837-0.829*=0.008 (0.9%) perfM2=perfRRperfLR .907-.829*=0.078 (9.4%) perfF=perfRFperfLR 0.820-0.829*=-0.009 (-1.1%) perfF1=perfLASSOperfLR - (MCC and F1 not calculated) for LASSO perfF2=perfRRperfLR 1.043-.829*=0.214 (25.8%) Note. Results are same as accuracy optimization = *. MCC and F1Test could not be performed = . 98 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Table 25 Performance All-Cause Urgent Care Indices Index Result perfA=perfRF perfLR 1.621-1.639=-0.018 (-1.1%) perfA1=perfLASSOperfLR 1.631-1.639-0.008 (-0.5%) perfA2=perfRRperfLR 1.347-1.639=0.292 (-17.8%) perfM=perfRFperfLR 1.665-1.639=0.026 (1.58%) perfM1=perfLASSOperfLR * perfM2=perfRRperfLR 1.665-1.639=-0.063 (-3.84%) Note. Results are same as accuracy optimization= *. 99 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 100 Figure 1. A Diagram of IHI Model for Improvement Showing Plan-Do-Study-Act process. Taken from How to Improve, by Institute for Healthcare Improvement, n.d. (http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx). Copyright 2019 by John Wiley and Sons. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 101 Figure 2. Representation of the Tradeoff Between Flexibility and Interpretability Using Different Statistical Learning Methods. Taken from Introduction to Statistical Learning: With Applications in R (p. 25), by G. James, D. Witten, T. Hastie, and R. Tibshirani, 2013, New York, NY: Springer Nature. Copyright 2013 by Springer Nature. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 3. Ridge Regression Coefficient Estimates for Each Value of . Taken from Introduction to Statistical Learning: With Applications in R (p. 216), by G. James, D. Witten, T. Hastie, and R. Tibshirani, 2013, New York, NY: Springer Nature. Copyright 2013 by Springer Nature. 102 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 4. Box Plot of Age by Center. Copyright 2020 by SAS 9.4. 103 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 5. Relapse ROC Curve Model Comparison Optimized for Accuracy. Copyright 2020 by R. 104 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 6. Relapse Gain Curve Model Comparison Optimized for Accuracy. Copyright 2020 by R. 105 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 7. Relapse ROC Curve Model Comparison Optimized for MCC. Copyright 2020 by R. 106 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 8. Relapse Gain Curve Model Comparison Optimized for MCC. Copyright 2020 by R. 107 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 9. Relapse ROC Curve Model Comparison Optimized for F1. Copyright 2020 by R. 108 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 10. Relapse Gain Curve Model Comparison Optimized for F1. Copyright 2020 by R. 109 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 11. All-Cause Urgent Care ROC Curve Model Comparison Optimized for Accuracy. Copyright 2020 by R. 110 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 12. All-Cause Urgent Care Gain Curve Model Comparison Optimized for Accuracy. Copyright 2020 by R. 111 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 13. All-Cause Urgent Care ROC Curve Model Comparison Optimized for MCC. Copyright 2020 by R. 112 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 14. All-Cause Urgent Care ROC Curve Model Comparison optimized for MCC. Copyright 2020 by R. 113 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 15. Relapse Model Measures Comparison Including All Optimizations (Accuracy, MCC, F1). Copyright 2020 by R. 114 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 16. All Relapse Model ROC Curves Including All Optimizations (Accuracy, MCC, F1). Copyright 2020 by R. 115 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 116 Figure 17. All-Cause Urgent Care Model Measures Comparison Including All Optimizations (Accuracy, MCC). Copyright 2020 by R. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 18. All-Cause Urgent Care Models ROC Curves Including All Optimizations (Accuracy, MCC). Copyright 2020 by R. 117 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 118 Appendix A IRB Approval Dartmouth College Dartmouth-Hitchcock Medical Center COMMITTEE FOR THE PROTECTION OF HUMAN SUBJECTS CPHS.Tasks@Dartmouth.edu 603-646-6482 SOCIAL, BEHAVIORAL, and NON-CLINICAL RESEARCH PLAN CPHS template v. 10/18/2016 Please complete: CPHS# PI: Brant Oliver (brant.j.oliver@dartmouth.edu) Important Note: The CPHS Department (Chair & Scientific) Review Form is required with this application. Find the form in the RAPPORT Library or on the CPHS Website. Respond to each item, even if to indicate N/A or not applicable Attach and/or upload this form as your Investigator Protocol in Rapport If you are completing this form on a Mac, indicate your answer to any checkboxes by bolding or highlighting, or by deleting any incorrect options. 1. Introduction and Background Historically, MS care has been studied and improved at the basic science, individual and population levels of analysis. However, in the new era of healthcare reform these approaches, while necessary and important, will no longer be sufficient. A paradigm shift towards the inclusion of systems-level approaches will be required to study and improve MS care and to demonstrate its value. This new focus derives from three critical developments. First, the IOM reports1 on quality and safety deficiencies in the U.S. healthcare system and the IHI Triple Aim2 have called for a new systems-oriented focus and continuous improvement culture. Second, Wennbergs seminal research on geographic variation,3 which established the Dartmouth Atlas of Health Care,4 demonstrated that local practice culture and patterns can displace evidence-based care and influence unwarranted utilization and increased costs.5-6 Finally, the Affordable Care Act is driving a shift from productivity to systems-level value-based STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 119 reimbursement (Accountable Care).7 It follows that a new culture of continuous quality improvement (CQI) will be required to optimize quality and demonstrate the value of MS care at the systems level. Quality Indicators in Multiple Sclerosis There has been some discourse regarding quality indicators in MS care. Cheung et al. (2010)15offer a substantive matrix of recommended quality metrics developed via a modified Delphi process which are heavily oriented towards clinical outcomes and related process measures (Figures 1 and 2). These span thirteen symptom specific clinical outcome domains (e.g. depression, spasticity, falls, etc.) and nine process specific domains referred to as General Health Domains of MS Care (e.g. patient education provided at time of diagnosis, provision of community and social resources, etc.). Recent work by the AAN (unpublished draft for public comment)16 (Figure 3) features a more concise yet similar collection of clinical and process measures focused on the provision of key clinical services and outcomes (e.g. falls screening, falls follow-up, depression screening, etc.), but also includes a functional health (quality of life) category. While encouraging, this work falls short of enabling the global measurement of quality, cost, and value in MS care. For example, cost, utilization, patient experience and satisfaction are not included in any of the current discourse, and disease modifying therapy (DMT) metrics and functional health measures are underrepresented compared to symptom management and clinical care process metrics. Additionally, there is no representation of structural resource measures. These limitations present challenges for cost-effectiveness analyses and the global assessment of the quality and value of MS healthcare services delivery. Towards Balanced Measurement of Quality and Value in MS Donabedians Structure, Process, Outcomes conceptual framework is well-established and often utilized in healthcare improvement science and can provide a strong foundation for a more comprehensive assessment of system performance in MS healthcare delivery.17 Structural measures include staff, facilities, and resources- the inputs or resources utilized to drive processes and outcomes in system. Process measures include practice patterns, e.g. MRI utilization, percentage of patients referred to social services, etc., very similar to many of the process measures listed in Figures 1-2. These measures represent how available structural resources are utilized by the system to generate desired outcomes. Metrics in this domain could span beyond those currently recommended to include a more detailed focus on disease modifying therapy (DMT) treatment initiation and access. Outcome measures represent the result of the employment of structural resources and processes to generate desired results. Outcome measures could include many of the metrics in Figures 1-2 such as depression status, annualized relapse rate, etc. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 120 Metrics in this domain could span beyond those currently suggested to include a more detailed focus on DMT treatment related outcomes, such as employment and social security disability status. Augmenting the Donabedian framework with Nelsons Balanced Measures Clinical Value Compass framework can further specify process and outcomes measurement.18 Nelsons model has four categories: (1) clinical outcomes (e.g. relapse rate, MRI status, symptoms); (2) functional health (e.g. self-efficacy, quality of life, social security disability status); (3) patient experience and satisfaction; and (4) utilization (e.g. MRI utilization, appointment frequency, ED utilization, etc.). Categories 1-3 are considered to be quality measures, and category 4 represents cost/utilization. Use of Nelsons framework can enable calculations of value indices (Quality/Cost).18-19 The proposed study includes the application of this expanded conceptualization of quality, cost, and value measurement utilizing a hybrid combination of the Donabedian and Nelson frameworks. Figure 4 demonstrates how these frameworks can be utilized to fit hypothetical examples of current metrics given in Figures 1-2, as well as additional measures which the current literature does not include. Driving Systems Level Improvement Three critical elements are required for improving the quality and value of MS care: (1) systems performance focus; (2) collaboratives; and (3) improvement coaching. Systems focus refers to a unit of analysis that is aggregated at a higher level than that of the individual clinician (e.g. physician, nurse practitioner, etc.) and which is focused at the level of the service delivery system (e.g. clinic, department, hospital), but at a lower level than that of epidemiological studies of populations (e.g. all MS patients in the United States). This level of analysis is a new focus in MS, which historically has focused on bench science, individual level, or population level analyses. While the recent development of quality metrics in MS is encouraging, its focus on individual clinician (e.g. physician, nurse practitioner) performance is unlikely to generate improvement on its own. The IOM reports (discussed previously) have established that individual efforts to work harder or work better are usually not enough to generate effective changes without incorporating a system level improvement focus. Additionally, Wennbergs work on local practice culture and geographic variation (also discussed previously) has demonstrated that without the use of benchmarking, transparency and system level improvement efforts, local practice trends remain staunchly resilient, often preventing the timely adoption of evidence based practices and stalling improvement efforts. This suggests that without the inclusion of system-level improvement efforts, the well-intentioned creation of MS quality metrics may only succeed in making hard working clinicians frustrated as they labor in vain to meet expectations that are impossible to reach. Improvement collaboratives have been shown to facilitate system-level performance improvement20-21 because they can facilitate benchmarking and transparency of system-level performance, accelerate the rate of learning, and motivate rapid cycle STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 121 improvement efforts. Collaboratives are defined as a group of two or more clinical care delivery systems which are focused on the same population and care delivery type (e.g. multiple sclerosis) that agree to share aggregated system level performance data, conduct benchmarking, and engage in learning and improvement together to enable accelerated improvement for all participating sites. The Cystic Fibrosis Foundation Learning and Leadership Collaborative (CFF LLC) 11 is an example of a systems-level improvement collaborative. The proposed study includes a small scale improvement collaborative design similar to that of the successful CFF LLC model and follows the general design recommended by the Institute for Healthcare Improvement (IHI) Breakthrough Series.22 Improvement coaching facilitates, guides, and organizes improvement efforts within healthcare delivery systems. Attempts to support teams in health care improvement have been reported in the literature for several decades and include supportive roles such as coaches, facilitators and helpers.23-27 Coaching actions include exploring the context where the team provides care and services, building relationships and communication processes with the improvement team and leaders, offering helping actions to support making improvements and reinforcing the improvement process by providing technical training and support. According to Gustafson (2013), coaching and improvement collaborative components are equally effective in achieving desired clinical outcomes improvement, and combining them is synergistic and additive.28 The proposed study recommends the utilization of an experienced improvement coach who has successfully guided improvement teams in a variety of contexts, including the CFF LLC and others, in combination with an improvement collaborative model. Mesosystem level interaction of microsystems in MS disease modifying therapy (DMT) access A priority that is shared by persons with MS, clinicians, industry, suppliers, and insurers is timely supply and sustained adherence to DMT (DMT access), which is well established as the basic fundamental backbone of MS care.40-45 A related critical aspect annual surveillance with brain MRI, which is utilized to monitor MS disease status and detect progression.46 Current discourse on quality measures in MS addresses DMT and MRI to some degree, but not with specificity.15-16 It follows that in order to address system level quality of MS care using a CQI collaborative, DMT and MRI utilization should be an initial process performance focus, and that the systems pathway contributing to DMT access be studied. Clinical microsystems concepts and terminology are helpful to employ to articulate this approach. Clinical microsystems represent the smallest possible unit of frontline healthcare service delivery and are the focus point for front-line healthcare improvement work.9, 19 In MS care, each MS clinical care system (MS clinic or MS center) is a clinical microsystem. The second order of systems is known as a mesosystem, 19 which is defined as two or more interacting microsystems engaged in a shared performance purpose. In the context of DMT access there are three microsystems involved: (1) the MS clinic; (2) the specialty pharmacy; and (3) the insurance provider. Together these three microsystems comprise a DMT Access Mesosystem (Figure 5). There has not yet been any formal systems level academic or practical study done on this pathway from a healthcare STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 122 improvement science perspective. The current state of interaction of these three microsystems appears to be loosely coupled and highly fragmented, with very little shared understanding between microsystem units or focus on collaborative mesosystem level performance. This may result in DMT access barriers at multiple levels (clinic to pharmacy, pharmacy to insurer, etc.). Although the primary focus for the MSCQI Collaborative is on MS clinical microsystems performance improvement, CQI interventions based in those microsystems aimed at improving DMT and MRI performance will need to include the entire mesosystem, including specialty pharmacies and insurers. Rationale Systems-based CQI approaches, such as Lean/Six Sigma8 and Clinical Microsystems9 are currently utilized in some healthcare settings across the United States. Regional and national CQI collaboratives utilizing these improvement methodologies have demonstrated significant results. The Northern New England Cardiovascular Network (NNE) Disease Study Group has utilized a shared data registry and QI methods to reduce morbidity and mortality across cardiac surgery centers in the northeastern U.S.10 On an even larger scale, over 110 Cystic Fibrosis Foundation (CFF) centers have participated in national level QI collaboratives which utilize a shared systems-level registry and QI methods. The CFF collaboratives have reduced mortality, improved life expectancy, reduced morbidity, and improved a number of process quality indicators.11-14 Data registries such as NARCOMS and the Slifka Longitudinal MS Study, have made important strides investigating MS care at the population level of analysis. Unfortunately, these registries are unable to conduct systems-level analyses, and there has not yet been investigation at the systems level regarding geographic variation, quality, and value of MS care. Additionally, there has not yet been an effort of any kind in MS including the use of improvement collaboratives aimed at bettering MS related outcomes, processes, quality or value. This study will establish the first QI collaborative for MS care in the US and will test the effects of QI interventions on selected priority performance indicators. 2. Objectives and Hypotheses Objective This is a 3 year study which aims to establish a CQI collaborative of four (4) MS centers/clinics (microsystems), to gather and benchmark systems-level performance across sites and for the collaborative as a whole, and then to study the effect of CQI interventions on improving system-level performance outcomes across microsystems. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 123 Specific Aims 1. To establish systems-level performance indicators by obtaining quarterly performance measures, aggregated by MS center/clinic (microsystem) and the entire MSCQI Collaborative. 2. To conduct studies of variation in performance across microsystems and to utilize benchmarking analyses to identify top performers. 3. To study the comparative improvement of selected primary process performance indicators (DMT and MRI process measures) over a 3 year period (Years 2-3) in microsystems receiving IHI Breakthrough Series CQI interventions versus those not receiving CQI intervention. Primary Endpoint The primary endpoint for this study is percentage of eligible MS patients on disease modifying therapy (DMT access), which is operationally defined as the total number of eligible patients on DMT/the total number of patients seen per quarter at a participating center for whom DMT is an appropriate treatment option. Secondary Endpoints This study will employ a balanced measures conceptual framework (Nelson et al 2004) that is commonly utilized in healthcare quality improvement to measure quality and value at the systems level. From this framework, measures from four functional domains will be employed: (1) clinical process and outcome measures; (2) functional health measures; (3) patient experience measures; and (4) cost and utilization measures. Secondary objectives are listed below by category: 1) Clinical Outcomes: Depression (PHQ-9), relapse rate, Patient Determined Disease Steps (PDDS). 2) System level (MS center level) aggregate Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) scores will be reported quarterly by participating MS centers. 3. Study Design Describe all study procedures, materials, and methods of data collection: This is a two-part prospective study to be conducted over 3 years with option to extend to 5 years. In the first part (Year 1), we will gather baseline performance data from participating MS clinics (microsystems), create a combined MSCQI systems-level database, and conduct analyses of performance variation and benchmarking (Specific Aims 1 &2). In the second part of the study STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 124 (Years 2-3), we will investigate the effect of CQI intervention on primary endpoints and selected secondary measures (Specific Aim 3). Benchmarking and Longitudinal Performance Monitoring: Years 1-3 Specific aims 1 and 2 will be addressed by quarterly benchmarking of all primary and selected secondary endpoint measures across microsystems throughout the entire duration of the study. Performance data will be aggregated by microsystem and by quarter and submitted electronically via an encrypted and password protected mechanism from each site to the MSCQI Hub Site. No personal identifying information will be collected. Quarterly data collected from sites will be analyzed to create quarterly benchmarking reports for each microsystem and performance reports for the MSCQI collaborative as a whole (basic example of a simple benchmarking data display is given in Figure 8). Figure 8. Example of a site hypothetical benchmarking report (quarterly MRI utilization) CQI Step-Wedge/Dynamic Clinical Trial: Years 2-3 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 125 The CQI intervention phase of the study will begin in Year 2. We will employ a step-wedge design (also known as a Comprehensive Dynamic Trial).29-30 Compared to a standard RCT or cluster randomized design, the step-wedge design can allow for additional comparisons within microsystems pre- and post- CQI intervention, can allow all participating microsystems to receive CQI interventions, and can better accommodate smaller sample sizes (Figure 9a-c). Figure 9a. Step-Wedge/Dynamic Clinical trial (3 Year Trajectory) Benchmarking IHI IHI Breakthrough Breakthrough Series CQI with Series CQI Improvement with Coaching Improvement Coaching (Year 1-3) Center 1 Center 2 Center 3 Center 4 X X X X (Year 2) (Year 3) X X X STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 126 CQI Intervention The improvement intervention employed in this study is an IHI Breakthrough Series CQI intervention including an improvement collaborative and professional improvement coaching. IHI Breakthrough Series CQI Intervention with Professional Improvement Coaching This intervention will be facilitated by clinical teams at participating sites under the guidance of a professional improvement coach utilizing the IHI improvement collaborative model. A hybrid adaptation of the Cystic Fibrosis Foundation (CFF) Learning and Leadership Collaborative (LLC) general structure and process as described by Godfrey and Oliver (2014)11 and the IHI Breakthrough Series improvement collaborative model22 will be utilized. One center will be randomized to this intervention in Year 2. The selected center(s) for the CQI intervention will form an improvement team consisting of members from the clinic itself. Each team will be advised by the professional improvement coach and instructed in basic improvement methods via a structured curriculum including didactic instruction, supervised application, networking and support provided through on-site meetings, online webinars, and coaching telephone calls (Appendix A). Each improvement team will conduct a system level context assessment of its clinical microsystem9 with a primary focus on the DMT access and brain MRI monitoring performance and a secondary focus on a selected secondary measure, e.g. patient satisfaction. With guidance from the improvement coach, teams will then formulate small scale interventions and associated real-time monitoring measures aimed at improving process performance for the primary endpoint measures (DMT and MRI). Teams will then pursue rapid cycle implementation, testing, and modification using repetitive Plan-Do- Study-Act (PDSA) cycles.35 Unlike large scale implementation science approaches which follow slower longitudinal trajectories and are more rigidly designed, rapid cycle improvement science approaches such as proposed here are more flexible, adaptive, and shortterm. For example, a typical PDSA cycle lasts 2-4 weeks and a typical healthcare improvement trajectory has multiple, successive PDSA cycles, with each cycle adapting and improving upon the last. Successive real-time modifications in the improvement strategy STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 127 will be made at the microsystem level by improvement teams based on measured performance feedback. Feedback will be provided via real-time Statistical Process Control (SPC) 38-39measurement (example in Figure 10) conducted directly by improvement teams on the front-line during improvement work and via bi-annual benchmarking reports. Year 3 Intervention The Step-Wedge/Dynamic Clinical Trial study design will allow for a flexible determination of study exposures in Year 3, which will allow for randomization of a second site to the IHI Breakthrough Series intervention(Figure 9) Year 2 Optional Readiness and Team Assesment Sub Study The sub study will try to understand the readiness of a clinical improvement team prior to and during improvement activities, including knowledge of quality improvement, systems level variation, readiness to learnand team behaviors (psychology) enabling or disrupting improvement activities. The sub study aims to investigate capability and readiness for engagement in improvement work with the collaborative and to identify potential barriers and facilitators to improvement across the different centers participating in the study. It will complement the MSCQI study focus on system level variation and effectiveness of the QI intervention by providing a descriptive study of the longitudinal readiness and capability of sites to engage in QI, as well as how these characteristics mature over time, We will employ qualitative interview and quantitative survey instruments designed to assess system and team readiness and capability for improvement work. MS Center team members at sites participating in the MSCQI collaborative will be offered the opportunity to participate in this study during MSCQI study quarterly site visits beginning in Fall 2018 following IRB review and approval. No patient participation or access to electronic medical record information is required. Data will be collected from participating MS center teams at quarterly site visits. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 128 Sub Study Design Several assessments/tools will be utilized to gain an understanding into the knowledge a team has of Quality Improvement, the teams readiness to learn, and behaviors. The assessments start at all consenting sites during the first quarterly site visit in Year 2 of the MSCQI study and include three time points (beginning of Year 2, beginning of Year 3, and end of Year 3 at the conclusion of the MSCQI study). The assessments will be administered to administrators, staff and providers only. It will not be administered to patients or families and therefore there are no human subjects involved in the study. The assessments will cover four areas; Knowledge, Readiness to Learn, Behaviors and Team Progress Knowledge The Quality Improvement Knowledge Assessment (QIKat) to assess the clinics staff and providers QI knowledge 40 This tool will be administered to all clinics staff and providers of all of the participating clinics (randomized and control) sites at all three time points. The survey will teak each participant approximately 5 minutes to complete. A Survey Monkey tool will be used to administer the survey (see Appendix E). Readiness to Learn About QI Readiness will be assessed for system leaders and improvement teams at all three time points. Leaders: The clinic leaders readiness to learn is a critical factor in influencing leadership tendency to create conditions for successful improvement for improvement teams. Leader readiness will be assessed utilizing a modified tool How Groups Learn Continuously (London, M., 2007). The tool consists of semi-structured questions, with Yes/No and free text options. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 129 This tool will be administered utilizing a Survey Monkey at all three time points. It will be administered to the clinic leaders only as either a direct interview or a Survey Monkey link to the survey. It will take approximately 5 to 7 minutes to complete depending on the comments made by the participants (see Appendix F). Teams: The clinics staffs and providers readiness to learn will also be assessed utilizing a similar tool modified from How Groups Learn Continuously. This tool also consist of semi-structured questions, Yes/No and free text options. This tool will be administered to all of the clinic staff and providers utilizing a Survey Monkey tool at all three time points. This survey will take the participants approximately 5 minutes to complete (see Appendix G). Behaviors: We will evaluate MS Center teams randomized to QI interventions during the MSCQI study, utilizing a structured qualitative assessment tool to describe and characterize behaviors/actions that may have an effect on improvement activities and their outcomes. This is an observational tool and will be administered by the core research team on the teams randomized to the coach supported QI intervention only. To correct for observer bias, at least two observers will utilize the tool to evaluate each team and reconcile by consensus any divergent assessments. The output of the evaluation is to describe categories of behaviors that may enable or disrupt QI activities in the improvement team, and to describe how these behavior characteristics vary across QI teams and change over time. The evaluation will take place during improvement team meetings and/or quarterly site visits during the MSCQI study (see Appendices H, I, and J). Team Progress STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 130 MSCQI sites randomized to QI intervention will undergo brief assessments of quality improvement progress utilizing the IHI Assessment Scale for Collaboratives (http://www.ihi.org/resources/Pages/Tools/AssessmentScaleforCollaboratives.aspx). This assessment will be completed monthly by each team that has been randomized to the coach supported QI intervention and by the coach, and the results compared. The tool will take approximately 5 minutes to be complete. A Survey Monkey will be used to administer the assessment. (see Appendix K). Sub Study Analysis This study utilizes a descriptive prospective cohort design. Results will be analyzed at the site level with comparison across participating MSCQI sites and compared to the aggregate as a whole. There will be 3 to 12 possible time pointes depending on the survey type over the length of the research. Descriptive statistics will be utilized to describe the general performance levels for quantitative measures. We will also perform a correlational analysis between the different measures to assess for interrelationships between these measures. We will use analysis of variance (ANOVA) to assess longitudinal change over multiple time points for these measures to evaluate if statistically significant changes in performance occurred during the course of the study. We will use thematic analysis to develop core themes from the qualitative data collected to organize into potential barriers and facilitators as well as an assessment of overall readiness to engage in improvement work. Sub Study Progress Monitoring Participation in the sub-study is at the discretion of each participating center/site and a centers decision to participate in the sub-study will not affect the conduct of the parent (main) study in any way. For the sites participating in the sub study, there will be monthly and quarterly opportunities for the core research team to interact with the participating clinics. At each monthly MSCQI collaborative webinar, the group will review the status surveys completed and outstanding. During the quarterly site visits, the core research team will have the opportunity to review assessments and work to validate data. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 131 Additionally there will be quarterly reviews with the supervising investigator to conduct integrity checks and review the overall study progress and adherence to the approved study protocol. Study Population There are two levels of participation in this study: (1) system level administrative; and (2) individual level clinical. System Level Administrative: The first level of participation is at the system level. It requires the reporting of de-identified administrative system level data aggregated to the system level by the four participating centers. This data will also include covariates such as payer mix and other relevant variables which will be used to adjust between centers. The measures collected for this level of the study will be described subsequently in section 9. This level of the study does not include the collection or use of protected health information (PHI) and will not require informed consent Individual Level Clinical: The second level of the study is at the individual level. It requires individual patients to complete self-report questionnaires and allow access to their medical records to abstract individual level data. This level of the study will include access to PHI, and will require written informed consent. All data will be de-identified and aggregated to the center level prior to use in the data analyses for this study. Inclusion Criteria To be eligible to participate in the Individual Clinical Level portion of this study, candidates must meet the following eligibility criteria. 1. Documented diagnosis of multiple sclerosis (MS) 2. Age 18 years or older STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 132 3. Ability to understand the purpose and risks of the study and provide signed and dated informed consent and authorization to use protected health information (PHI) in accordance with national and local privacy regulations. Exclusion Criteria Candidates will be excluded from study entry if they are unable or unwilling to provide informed consent. Data will not be abstracted from medical records for these patients for the study. Measures and Data Sources This study will employ a comprehensive system level performance measurement strategy featuring quarterly reporting, semiannual benchmarking data feedback to sites, a one year baseline assessment period, and a two year evaluation of QI intervention effects. Three data sources will be utilized for this study: (1) center level administrative data; (2) individual level data abstracted from medical records and subsequently de-identified and aggregated to the center level; and (3) individual self-report data connected by electronic data capture (EDC) methods that will subsequently be de-identified and aggregated to the center level. Variables for the study are given in Table 2 below and are listed by Nelsons domains (clinical outcomes, functional health outcomes, patient experience, and utilization/cost). Table 2 lists data sources, the priority of the data collection method (core versus exploratory) and the type of endpoint (primary versus secondary). Clinical and functional health measures, which are derived from medical record abstraction (MRA) or self-report questionnaires, require informed consent whereas patient experience, utilization, and covariate measures, which are derived from administrative data, will not require consent (see Table 2a). Table 2. MSCQI Collaborative Measures Category Measure(s) Type Source Frequency IC STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 133 Population Characteristics Race/Ethnicity, Socioeconomic status Primary language Marital status Education Employment Co-morbidities, Concomitant meds, Current/previous DMTs Supplemental PRO Portal Annually Yes Clinical DMT Access Core Yes (% on DMT) (Primary) Medical Quarterly record abstract ion (MRA) Clinical % MRI in past year Core MRA Quarterly Yes Clinical Relapses Core MRA & PRO Portal Quarterly Yes Clinical PHQ-9 Core PRO Portal Quarterly Yes Clinical Vitamin D Level Core PRO Portal Annually Yes Clinical PDDS Core PRO Portal Semi-Annually Yes STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Functional Health Neuro-QoL 134 Core PRO Portal Semi-Annually Yes Core PRO Portal Quarterly Yes Ability to Participate in Social Roles and Activities Lower Extremity Function Upper Extremity Function Stigma Satisfaction with Social Roles and Activities Sleep Disturbance Communication Functional Health Neuro-QoL Anxiety Cognitive Function Functional Health WPAI Core PRO Portal Semi-Annually Yes Functional Health PROMIS Fatigue SF Core PRO Portal Quarterly Yes Patient Experience BAI Core PRO Portal Quarterly Yes Patient Experience CG-CAHPS Core Admini strative Annually No My Health Confidence STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 135 Your Opinion Matters Patient Experience TSQM-9 Core PRO Portal Semi-Annually Yes Utilization MRI Core Admini strative Quarterly No Utilization Office Visits Core Admini strative Quarterly No Adherence ED Visits Hospitalization Table 2a. Descriptive & Covariate Measures: Collected Annually by Center (Administrative) Category Measure Information Population Core Demographics % Female Gender Population Core Demographics % RRMS MS Disease Type Population Core Demographics Age distribution Age Region State Geography Payor Mix % Private Insurance Insurance coverage Payor Mix % Government Insurance Insurance coverage STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 136 Providers (MD, DO, NP, PA) #FTE Capacity Nursing (RN, LPN) #FTE Capacity Social Worker/Mental Health #FTE Capacity Rehab Providers (PT, OT, etc.) #FTE Capacity/Access Provider Visits Available Average #visits available per month Capacity/Access (Potential) Certification NMSS or CMSC certified Recognition as Center of Excellence for MS care Clinical Setting Academic, Community Hospital, Private Practice type Clinical Setting Urban, Rural Community setting Data Collection and Processing Pathway Data to be collected in this study will be abstracted quarterly in de-identified form via data downloads from electronic medical records and administrative records from participating centers and from the PRO Portal as previously described (see Figures 6-7). EMR Data will be entered by each site into a secure Redcap database maintained on the Dartmouth Hitchcock server. No PHI will be entered into this database. All PHI will remain at the specific sites. Once the data has been aggregated into the Redcap database it will be submitted in a highly encrypted, aggregate form to a MSCQI hub site database along with the PRO data for management and analysis. Jefferson School of Population Health (JCPH) will provide data management and analyses for this study. A hub site database will be constructed and managed by JCPH using its own resources and secure firewalls. Security standards for JCPH will be STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 137 assured at computing levels required by Dartmouth Hitchcock Medical Center. Site sub-investigators and/or their designees, and the MSCQI research coordinator will facilitate data abstraction and secure transmission to the hub site database with assistance and oversight provided by the study methodologist and the principal investigator. Participating Sites and Key Personnel This study will be facilitated by four entities: (1) the core research team (MSCQI Hub Site); (2) an independent database management site (JCPH); (3) MSCQI sites (microsystems) and sub-investigators; and (4) the Research and Improvement Advisory Committee (RIAC). (1) MSCQI Hub Site and Core Research Team Personnel Leadership and management for the study will be centralized at the Dartmouth Population Health Collaboratory research hub site as previously described in Section 1. The core research team will consist of the principal investigator (PI), the methodologist, the professional improvement coach, and the research program manager/study coordinator. Principle Investigator (Brant Oliver, PhD, MS, MPH, APRN-BC, MSCN): See Section 1. Co-Investigator/Improvement Coach (Randy Messier MT, MSA, PCSH CCE): Mr. Messier will serve as improvement coach and co-investigator for this study. He is an experienced professional improvement coach with management and administrative training and is also certified in Patient-Centered Specialty Home (PCSH CCE). He has served successfully as a professional improvement coach for many hospitals and improvement collaboratives in the United States and Canada, including the CFF LLC national collaborative (during which time he has worked closely with Dr. Oliver)11 and multiple state-level improvement collaborations in Vermont, including recent work in the Optimizing Laboratory Testing Collaborative . STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 138 Research Program Manager and Study Coordinator (Amy E. Hall, MS): Ms. Hall will serve as research program manager and study coordinator. She has extensive experience in program management, study coordination and data abstraction. Her duties will include coordination between the hub site and participating MSCQI sites, regulatory and budgetary coordination, logistics, and data abstraction (including travel to MS centers for data abstraction). (2) Independent Data Management and Data Analytics Center (Jefferson College of Population Health Innovation) The MSCQI Collaborative study will maintain its research database at a site independent of the funding source and the study PI. This is of particular importance in industry-funded and investigator-initiated studies to reduce actual and/or perceived risks of bias in design and conduct of the study and analysis of the data. For this reason it is proposed that Jefferson College of Population Health Innovation (JCPH) provide data management for this study and that this study and that this aspect of the study be overseen by Alexis Skoufalos, EdD. The MSCQI hub database will be constructed and managed by JCPH. (3) MSCQI Sites (Microsystems) and Sub-Investigators Four (4) multiple sclerosis centers/clinics (microsystems) from the eastern United States will participate in the MSCQI collaborative study. Oversight for each participating site will be provided by a study sub-investigator who will serve as a site principal investigator. Site investigators are experienced MS clinicians and/or researchers who have worked with the study PI (Dr. Oliver) in the past and have the capability to manage and coordinate activities at their respective sites to accommodate the needs of the proposed study, including data abstraction and reporting, and engagement in CQI efforts during the intervention phase of the study. Total panel size for all four sites combined is approximately 5,600 MS patients (N=5,600). Participating sites represent urban and rural settings, academic and private practice contexts. MGH Multiple Sclerosis Clinic (Site Investigator: Eric Klawitter, MD, M.Sc.): The MGH MS Clinic is an urban academic MS center in Boston, MA that is affiliated with the Partners Healthcare system. It follows approximately 1,000 MS patients and is heavily involved with MS clinical trials. Dr. Klawitter is a NMSS fellowship trained MS neurologist and research scientist with expertise in MRI imaging. He directs the MGH MS Clinic. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 139 University of Vermont (UVM) Multiple Sclerosis Center (University of Vermont Medical Center (Site Investigator: Andrew Solomon, MD): The UVM Multiple Sclerosis Center is a rural academic MS center in Burlington, VT which follows approximately 1,500 MS patients. It is affiliated with the University of Vermont (UVM) Medical School, and is heavily involved with clinical trials research. Neurology Associates Multiple Sclerosis Center of Greater Orlando (Site Investigator: Patricia Pagnotta, MS, APRN-BC, MSCN): The multiple sclerosis practice at Neurology Associates of Greater Orlando represents a large private MS clinic, following approximately 1,000 MS patients. The MS practice there has an extensive MS clinical trials research program paralleling similar programs in academic MS centers. Patricia Pagnotta is an experienced certified MS nurse practitioner in the practice and is closely involved with the MS research program. Concord Hospital Multiple Sclerosis Specialty Care Clinic (Site Investigators: Ann Caobt, DO and Jennifer Taylor, ARNP): The multiple sclerosis practice at Concord Hospital represents a small private hospital in a rural community. The clinic follows approximately 1200 MS patients. 4. Analysis Describe any qualitative tests and measures as well as quantitative methods: The data analytic plan for this study is intended to reflect, to the degree possible, the actual effects of system performance on population health. The analysis population will include all administrative data collected at the system level combined with all individual level data collected from all consenting individuals, aggregated up to the systems level. Statistical tests will be conducted using IBM SPSS Version 22 (Chicago, IL) and/or STATA (Statacorp Inc., College Station, TX) software with alpha set at p < .05. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 140 Prior to analysis, all responses will be examined for accuracy of data entry, missing values, and fit between their distributions and the assumptions of the statistical analysis including normality, homogeneity of variance, linearity, and colinearity. Violations of the aforementioned assumptions will lead to transformations (square root, logarithm, or inverse) to reduce skewness, reduce the number of outliers, and improve the normality, linearity, and homoscedasticity of residuals. Subsequently, specific descriptive and analytic examinations will be conducted by specific aim as described below. Specific Aim 1: To establish systems-level quality and value (i.e. cost-effectiveness, efficiency) performance levels by obtaining quarterly performance measures, aggregated by MS clinic (system) and the MSCQI Collaborative as a whole. Appropriate descriptive statistics will be conducted to describe basic system-level variation in primary endpoint measures (DMT and MRI) and selected secondary endpoint measures from each of the major balanced measurement domains (clinical, functional health, patient experience, and utilization). Measures will be adjusted based on appropriate system level structural and individual level demographic characteristics which demonstrate statistically significant relationships with the dependent variable in univariate analyses. Specific Aim 2: To conduct studies of geographic variation in performance across MS clinics (microsystems) and utilize benchmarking analyses to identify top performers. A hierarchical linear model (HLM) for performance levels is recommended for Aim 2. The HLM will incorporate the performance measures and the effects of geographic location. To assess the importance of geographic variables, three types of models will be compared: a model with individual level variables only; a model with system level geographic effects that do not interact with person attributes; and a full model, allowing for geographic level random effects that differ by site. Both of the primary endpoint measures and at least one of each category of quality measures will be included in benchmarking analyses: (1) clinical measures (e.g. EDSS); (2) functional health measures (e.g. Neuro-QOL); and (3) a patient satisfaction measure (CG-CAHPS). The Available STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 141 Benchmarks of Care (ABC) Benchmarking Method37 may also be utilized to benchmark variables that demonstrate substantive variation in key process or outcomes performance and to identify top performing sites by performance category. Specific Aim 3: To test the effect of rapid cycle CQI interventions in improving selected quality and value outcomes in participating sites Standard monitoring of all measures will continue throughout the intervention phase in Years 2-3 similar to that of Year 1. Longitudinal quantitative analyses, site specific real time assessments using statistical process control (SPC), and qualitative analyses can be conducted to determine improvement effect in intervention sites compared to controls and compared to each other. See sections 11.4-11.6 below for detailed descriptions of analyses for this specific aim Primary Endpoint Analysis This section is relevant to the analysis of Specific Aim 3 as previously discussed above. Longitudinal time series regression analyses will provide a robust assessment of overall intervention effects on DMT treatment access. DMT access will be treated as the DV and compared between baseline and quarterly intervention time periods during Years 2-3, overall and stratified by intervention and benchmark. We will use Chi-square tests for categorical data and Student t-tests for continuous data. We will use multilevel XTME Poisson regression clustering to the clinic/center level to calculate adjusted risk ratios (RR) with 95% confidence intervals (95%CI) of the selected DV measures between the intervention and baseline periods adjusting for important system level structural and individual level demographic characteristics. To demonstrate temporal trends in performance, we recommend interrupted time series analyses of quarterly performance rates adjusting for covariates in the Poisson model. Secondary Endpoints Secondary endpoints, as described previously, will be analyzed using similar methods to the primary outcome variable. Longitudinal time series regression analyses will provide a robust assessment of overall intervention effects on secondary outcome STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 142 variables. These variables will be treated as DVs and compared between baseline and quarterly intervention time periods during Years 2-3, overall and stratified by intervention and benchmark. We will use Chi-square tests for categorical data and Student t-tests for continuous data. We will use multilevel XTME Poisson regression clustering to the clinic/center level to calculate adjusted risk ratios (RR) with 95% confidence intervals (95%CI) of the selected DV measures between the intervention and baseline periods adjusting for important system level structural and individual level demographic characteristics. To demonstrate temporal trends in performance, we recommend interrupted time series analyses of quarterly performance rates adjusting for covariates in the Poisson model. Interim Analyses Two types of interim analyses will be conducted during the study: (1) benchmarking analyses (see Specific Aim #2 above); and (2) Statistical Process Control (SPC) analyses (see Specific Aim #3 above). These sections are repeated below. 1) Benchmarking Analyses (Specific Aim #2): Benchmarking analyses will be conducted semi-annually and results reported as feedback to sites to contribute to the overall function and improvement mission of the MSCQI Collaborative. A hierarchical linear model (HLM) for performance levels is recommended for Aim 2. The HLM will incorporate the performance measures and the effects of geographic location. To assess the importance of geographic variables, three types of models will be compared: a model with individual level variables only; a model with system level geographic effects that do not interact with person attributes; and a full model, allowing for geographic level random effects that differ by site. Both of the primary endpoint measures and at least one of each category of quality measures will be included in benchmarking analyses: (1) clinical measures (e.g. EDSS); (2) functional health measures (e.g. Neuro-QOL); and (3) a patient satisfaction measure (CGCAHPS). The Available Benchmarks of Care (ABC) Benchmarking Method37 can be utilized to benchmark variables that demonstrate substantive variation in key process or outcomes performance and to identify top performing sites by performance category. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 143 2) Statistical Process Control (Specific Aim #3): SPC measurement methods38-39 will be utilized by centers randomized to the IHI Breakthrough Series QI intervention. SPC analyses will be developed by front-line improvement teams during rapid cycle change interventions (PDSA cycles) to assess short-term, context- specific performance metrics and to adjust real-time CQI intervention efforts accordingly in order to maximize longitudinal improvement performance. An example of a SPC chart is given in Figure 10. Figure 10. A Statistical Process Control (SPC) Chart used in real time CQI work by improvement teams (for a site level process measure related to DMT access primary outcome) 5. Study Progress Monitoring Note: appropriate monitoring may include periodic assessment of the following: data quality timelines recruitment and enrollment Provide a description of the methods which will be used to determine the progress of the study, including periodic assessments of data quality, timelines, recruitment, and enrollment as appropriate: STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 144 Study monitoring will be conducted on two levels: (1) investigator level; and (2) via a Research and Improvement Advisory Committee (RIAC). Investigator Level Monitoring The Dartmouth research hub site will monitor overall study progress in real time with adherence to quarterly time points established by the study schema given below. This schema will provide the study timeline, which provides quarterly time points for data collection and transmission to the data analytics center (TBD), at which time a quarterly assessment of recruitment and enrollment will also be ascertained. The study investigator will meet regularly with the data analytics center at a minimum of once monthly for ongoing work on data collection and analysis, and this work will also include a quarterly assessment of data quality. Table 1. Study Activities Schema Start of Quarter Deadline for Deliverables Start-up, Contracting Internal Review Board (IRB) initial submission and reviews Annual Report to Funder Q0 - Q1 7/1/ 17 Q2 10/1 Q3 1/1/ 18 Q4 4/1 Q5 7/1 Q6 10/1 Q7 1/1/ 18 Q8 4/1 Q9 7/1 Q10 10/1 Q11 1/1/ 19 Q12 4/1 12/3 1 X 10/1 1/1 4/1 7/1 10/1 1/1 4/1 7/1 10/1 1/1/ 19 7/1 6/30 X X X X X X X STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Measures collected (All Sites) Benchmarkin g Reports to Sites CQI Intervention (Step-Wedge Design) Coaching Site Visits Coaching Calls with Teams Improvement Skills Webinars On-Site Sessions RIAC meetings X X X X X 145 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Research and Improvement Advisory Committee (RIAC) The RIAC will provide balanced advisement, transparency and monitoring of study activities by key stakeholders and technical experts. The RIAC will be chaired by a MS clinician or researcher selected by the Principal Investigator (Dr. Oliver), and will include at least one of each of the following additional members (also selected by the PI): (1) at least one person who has MS; (2) a scientific representative designated by the funder; (3) a healthcare quality improvement specialist and/or researcher; (4) a representative from a DMT specialty pharmacy; (6) a second MS specialist clinician or researcher; and (5) a representative from an insurer/payer (if available). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 146 The RIAC will meet biannually with the Principal Investigator and other core research personnel as needed to discuss study progress and recommendations for the ongoing conduct of the study. RIAC members will be bound by a confidentiality agreement and must complete online CITI training for human subjects research. The RIAC will be of particular importance in assuring an appropriate practical balance between stakeholder needs, ethical standards, and risk of stakeholder bias in industry-funded and investigator-initiated research focused on the assessment and improvement of systems-level performance in MS centers/clinics. Maintaining this balance will be critical to the investigators and the funder in order to cultivate a high standard of credibility concerning the conduct of the study and the integrity of its methods and results. 6. Risks & Benefits Note: Risks may be physical, psychological, social, legal, economic, to reputation, or others. a. Describe any potential risks, their likelihood and seriousness: Because this study does not employ any invasive procedures or treatments, the risk to participants for participating in the study is assumed to be minimal and no greater than that engendered by participating in routine MS care. This study will include the collection of personal identifying information (PHI) which introduces associated risks, but these are estimated to be low as will be discussed in later sections addressing data security. Study procedures will not alter standard multiple sclerosis care for participants or influence the patient-provider interaction or relationship in any way. Data collected for this study will be managed by experienced personnel at a designated independent data analytics center (JCPH). Designated Data Analytics Center (JCPH) STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 147 Data to be collected in this study will be abstracted quarterly in de-identified form via data downloads or manual abstraction from electronic medical records and administrative records from participating centers and from the PRO Portal as described below and in data flow process Figures 6-7. Data will be submitted in aggregate form from the Dartmouth hub site to a MSCQI database for management and analysis. Jefferson College for Population Health Innovation will provide data management and analyses for this study. A hub site database will be constructed and managed by JCPH using its own resources and secure firewalls. Site subinvestigators and/or their designees, and the MSQCI research coordinator will facilitate data abstraction and secure transmission to the hub site database with assistance and oversight provided by the study methodologist and the principal investigator. JCPH was purposively recruited to provide research methodology and data analytics services for the MSCQI Collaborative study for the following reasons: (1) significant past history of working with academic institutions and multiple industry entities without a predominant interest or affiliation with any particular entity; (2) significant expertise in the organization and its leadership in population health research (including patient reported outcomes, quality of life research, and healthcare quality research); (3) capacity to serve as an independent data repository and data analytics entity functioning separate from the MSCQI Collaborative entities and the funding source, thus optimizing the integrity of data management and reducing threats to objectivity posed by stakeholder (funder, study site, etc.) interests. Patient Reported Outcomes (PRO) Portal Description The PRO Portal system (see Appendices C and D) will include a secure browser and device compatible system with the ability to register role based users, ability to deploy validated electronic surveys, develop surveys, a graphical view module containing descriptive statistics for real-time acquired data, a system dashboard containing user characteristics, and a timeline module to document key historical events for groups of participants. Participants will use system to enter health data, complete validated electronic surveys, document event driven events, i.e. MS relapse, concomitant medications, co-morbidities, etc. Investigators will login to review de-identified aggregated data in a graphical view module for aggregate de-identified data downloads for study-related data analyses. All PHI stored in the PRO Portal is 128 bit encrypted and fully compliant with current HIPAA standards (see Appendix STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 148 C). Access to encrypted PHI in the PRO Portal will be upon secure log-in with the operations system by authorized users only. Participants will not be identified by name in de-identified data used for study analyses or publications and data collected will be used for research purposes only. b. Confirm that risks to subjects have been minimized, by use of procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk: As discussed in Part A above, actual risks to participants are assumed to be minimal and no greater than routine multiple sclerosis care and because de-identified data will be collected, confidentiality risks due to study participation are minimized. Additionally, the use of a step-wedge research design allows for rigorous empirical investigation of the research question utilizing a robust set of measures but with a low time duration and limited number of participating MS Centers. Additionally, the study does not influence the patient-provider relationship or selection of multiple sclerosis medical treatments in any way. This combination of study design characteristics effectively minimizes participant burden and risk. c. Describe why all the risks to subjects are reasonable in relation to both anticipated benefits and the knowledge expected to be gained from the study: As described in Part A and B above, risks associated with study participation are anticipated to be minimal and no greater than that associated with standard multiple sclerosis care. The anticipated benefit of this study is the establishment of the first national quality improvement collaborative for multiple sclerosis in the United States and initial research on system level performance variation in multiple sclerosis care and the comparative evaluation of quality improvement interventions aimed at bettering multiple sclerosis care outcomes and care quality. This study could greatly benefit the multiple sclerosis care community by generating new knowledge about best practices and new approaches to improving system level performance in managing this very common and costly STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 149 chronic illness population. Finally, this study may also establish key initial findings demonstrating the feasibility of an improvement research collaborative for multiple sclerosis and a framework for subsequent research and development in this area. 7. Unexpected Events or Incidental Findings Note: It may be important to consider the potential for certain unanticipated events to occur, for example: finding an anomaly in a MRI discovering child abuse causing distress in interviews of a sensitive nature Describe potential events and provide a plan of action: Because this study does not employ any invasive procedures or treatments, the risk to participants for participating in the study is assumed to be minimal and no greater than that engendered by participating in routine MS care. However, throughout the course of the study, every effort must be made to remain alert to possible adverse events (AEs). If an AE occurs, the first concern should be for the safety of the subject. If necessary, appropriate medical intervention should be provided. All participants in the study will be given a study information sheet which will contain contact information for the study investigator. Any untoward or unfavorable medical occurrence in participants, including any abnormal sign, symptom, or disease, temporally associated with the subjects participation in the research, whether or not considered related to the subjects participation in the research, will be termed an adverse event and reported. Adverse events will be evaluated by the investigator or his medically qualified delegate to determine if the adverse event represents an unanticipated problem. Adverse events that meet the following criteria will be deemed unanticipated problems and reported to the IRB: The adverse event is unexpected STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 150 The adverse event is related or possibly related to participation in the research The adverse event suggests that the research places subjects or others at a greater risk of harm than was previously known or recognized 8. Deception Does any part of this study involve deception or withholding of information from participants? Yes No If Yes, provide an explanation which addresses the following: A description of the deception being used Why the deception is necessary A plan for debriefing, or providing subjects with the pertinent information after participation 9. Equitable Participant Selection a. Estimated number of participants at Dartmouth CPHS reviewed sites: A sample size of 2,000 participants (cumulatively from all four participating sites) is estimated based upon the proposed study measures and data analytic plan. Two of the participating sites (Orlando and MGH) have ceded IRB review to Dartmouth (representing approximately one half of the 2,000 total, or 1,000). The other two participating centers (University of Vermont and Concord Hospital, representing the other 1,000 participants, have elected to undergo separate IRB review by their respective institutions). STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 151 b. Provide a justification of the proposed sample size This study aims to investigate and improve real-life MS care delivery at a variety of MS centers (microsystems) across the United States. A convenience sample of four (4) MS centers/clinics (sites) that have expressed interest in participating in this study have been recruited, following approximately 5,600-6,000 persons with MS. Actual sample size (total number of participants required) for data analyses for main effects based on our proposed data analytic plan is determined by the final number of variables selected for inclusion in the final regression model for main effects analysis. This will be determined based on univariate analyses of significant factor contributors to outcomes for each of the component measures as well as factor analyses which will identify significant clusters of variables which can be represented by the most significant contributors in each cluster (thus minimizing sample size demand). Given that baseline data will be required to conduct univariate analyses and factor analyses to calculate the actual sample size for final main effects regression models, we used standard conventions for multiple regression modeling36 to estimate a maximum sample size for conducting primary analyses with acceptable power for final main effects regression models containing ten variables and repeated measures in a step-wedge research design. We will use this as our final modeling constraint, i.e. a maximum of ten highest priority variables for each main effects primary endpoint regression analysis. We estimate that a sample size of approximately 2,000 participants (500 per recruitment site) will be required for acceptable power for main effects analyses given these constraints. To meet the recruitment goal of 2,000 participants, we will need to maintain participation (at the system level) of approximately one third of the total available multiple sclerosis patient population receiving standard care from each of the participating MS centers in the research collaborative. We will not close the study to participation once the minimum N of 2,000 participants is achieved. Because this is a minimal risk, minimal patient burden study, we will allow all comers wishing to participate to do so in the study throughout its entire duration. c. Define the target population: STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 152 The target population for this study is characterized as adults diagnosed with multiple sclerosis who receive standard multiple sclerosis care at one of the four participating multiple sclerosis centers in the research collaborative and meet inclusion and exclusion criteria for study participation. d. Vulnerable populations Note: Certain populations are considered vulnerable to coercion and undue influence and are provided with additional protections when participating in a research study. Identify any of the below populations which you plan to recruit for this study. In addition, complete the form(s) linked with each population as necessary and upload on the Supporting Documents page in Rapport. Pregnant Women, Fetuses and Neonates Children People with impaired decision-making capacity The following populations may also be considered vulnerable to coercion or other undue influence: Prisoners People who are economically disadvantaged The elderly People who are illiterate or do not speak English Students and employees Describe any other potentially vulnerable population(s) and the additional protections provided to them: Not applicable. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 153 10. Recruitment Describe method(s) of recruitment. Associated advertisements and other materials to be used for recruitment should be uploaded to the Consent Forms and Recruitment Materials page in Rapport. Participants will be recruited consecutively at the time of clinic visits at participating MS centers. Research coordinators on site will offer participation by providing the informed consent (IC) form for review. The IC form will be available on the PRO data collection device (see Appendices C & D) which will be on a tablet computer. Paper versions of the IC form will be available for those wishing to have a paper copy. The PRO interface will allow for written informed consent to be completed via tablet computer. Individual level data will be collected for these participants who have given informed consent. Subsequent to enrollment, study staff and the Investigator will verify that participants are eligible per criteria. Upon confirmation of eligibility, participants will be assigned a registration number which will be used to identify data for each participant and link data derived from questionnaires to that abstracted from medical records, as well as to track which patients have given informed consent to participate. Only individual sites will maintain records linking medical record numbers to study identification numbers but these will not be transmitted to the hub site or be used in benchmarking analyses other data analyses for the study. Prior to transmission of aggregated quarterly data to the hub site by participating MS centers, all PHI will be removed save for the study identifier. Participants will be withdrawn from the study for any one of the following reasons: The participant withdraws consent or wishes to discontinue participation. The participant is unwilling or unable to comply with the protocol. A healthcare provider withdraws the participant from the study for medical reasons. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 154 Participants will notify the study staff in writing via standard letter or electronic communication of desire to discontinue participation at any time. 11. Informed Consent, Assent, and Authorization All forms discussed in this section should be uploaded to the Consent Forms and Recruitment Materials page in Rapport a. Please describe the consent and/or assent process, addressing the following: Who will obtain consent/assent from participants Where the consent/assent process will take place The timeframe for providing information potential participants about a study, having the consent form signed, and beginning study activities Any precautions taken to minimize the possibility of coercion or undue influence The forms which will be used as well as any aids used to simplify scientific or technical information How comprehension will be ensured Because PHI will be collected in this study, written informed consent will be obtained. Data will be de-identified prior to transmission to the study data analytics center and a highly secure PRO data collection system will be used. Participants will be offered participation and complete informed consent at the time of their regularly scheduled MS care appointments by study staff at participating MS Centers. We anticipate that the risk of PHI dissemination will be minimal and the actual risk of participation for this study is considered minimal and no greater than that of standard multiple sclerosis care. The study principal investigator may also be contacted with study related questions. Patients desiring not to participate may inform local site study staff or the principal investigator at any time to opt out of study participation. Those providing written informed consent will be able to participate in the study. b. Waiver(s) or alteration(s) may be requested for research that involves no more than minimal risk. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 155 Indicate requested waiver(s) or alteration(s) below. In addition, complete the corresponding section of the Waivers and Alterations Request Form and upload it to the Consent Forms and Recruitment Materials page in Rapport. For the informed consent process For the documentation of informed consent For the HIPAA Authorization to use and/or disclose PHI For a waiver of the requirement for medical record documentation 12. Compensation or Gifts Please describe any payments, gifts or reimbursements participants will receive for taking part in the study: No remuneration will be provided for participation in this study. 13. Privacy of Participants Note: Methods used to obtain information about participants may have an effect on privacy. For example: Consent discussions or interviews held in public which concern sensitive subjects or behaviors Observations of behavior, especially illicit behavior, in quasi-public settings Describe any activities or interactions which could lead to a breach of privacy and provide a plan to protect participant privacy: Data collected via the PRO mechanism carries a small risk to privacy because this data includes limited PHI elements. However, because data will be collected using a highly secure PRO mechanism and de-identified and numerically codified prior to transmission to the data analytic center for data analyses, there will be no record or information linking information gathered for STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 156 this study to the individuals providing that information. All other data collected for this study will be at the system (MS Center) level and will not link to any form of PHI. 14. Confidentiality of Data Note: Any person engaged in research collecting information about illegal conduct may apply for a Certificate of Confidentiality from the National Institute of Health. a. If disclosed, could any of the data collected be considered sensitive, with the potential to damage financial standing, employability, insurability, or reputation? No Yes If Yes, describe the data or information, the rationale for their collection, and whether a Certificate of Confidentiality will be obtained: Not applicable. b. Describe the safeguards employed to secure, share, and maintain data during the study, addressing any of the following which may apply: Administrative, i.e. Coding of participant data Physical, i.e. Use of locked file cabinets Technical, i.e. Encrypted data systems The data collection procedures in this study include the collection of data with aggregation to the system (MS Center) level and transmission of this data to the data analytics center in de-identified, aggregate form. The PRO platform used for this STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 157 study includes sophisticated data security including 128 bit encryption and password protection for data transmissions on database access for study personnel. The use of physical (non-electronic) records for this study will be minimal, and any physical records will be maintained in a double-locked environment, i.e. locked filing system in a locked office. Finally, coding of participant data will be conducted on a system level, i.e. data will be linked to a population of individuals rather than to individual participants, i.e. to one of the four participating MS Centers. c. Describe the plan for storage or destruction of data upon study completion: Data for this study will be maintained in de-identified, aggregate format, in a secure, encrypted, and password protected storage system with redundancy back up protection maintained by JCPH, the independent data analytics center. Data will not be maintained by any of the participating sites. After all activities and data analyses are completed for this study, study data will be maintained for the minimum period specified by IRB requirements and then will be permanently deleted. FIGURES Figure 1 List of Draft Quality Indicators for MS (Cheung et al., 2010) Figure 2. General Domains of MS Care (Cheung et al., 2010) Figure 3. Draft AAN Quality Measures Set (2014) Figure 4. Hybrid Donnabedian Nelson Framework Figure 5. Mesosystem interactions: MS clinic, pharmacies, and insurance Figure 6. PRO Portal relationships to DMT pathway and data processing Figure 7. Donabedian-Nelson measurement framework Figure 8. Hypothetical example of a benchmarking Feedback report to centers Figure 9. Step-wedge design randomization schemas STATISTICAL METHODS FOR MULTIPLE SCLEROSIS Figure 10. Example of a SPC chart for use in measurement of rapid cycle improvement Figure 11. Patient Reported Outcomes (PRO) Schedule APPENDICES Appendix A. IHI Breakthrough Series Collaborative and Coaching Support Activities Schema Appendix B: Figures 1-11 Appendix C: PRO Platform Technical Guide Appendix D: PRO Informed Consent Form Signature Page Example 158 STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 159 References: 1. Institute of Medicine (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. Available Online: http://www.iom.edu/~/media/Files/Report%20Files/2001/Crossing-the-QualityChasm/Quality%20Chasm%202001%20%20report%20brief.pdf. 2. Institute for Healthcare Improvement (2014). The IHI Triple Aim. Available Online: http://www.ihi.org/offerings/Initiatives/TripleAim/Pages/default.aspx. 3. Wennberg J, Gittleson A (1973). Small Area Variations in Health Care Delivery. Science; 14(182): 1102-1108. Available: http://www.dartmouthatlas.org/downloads/papers/Science_1973.pdf. 4. Trustees of Dartmouth College (2014). The Dartmouth Atlas of Health Care (Website). Available Online: http://www.dartmouthatlas.org/. 5. Welch WP, Miller ME, Welch GH, Fisher E, Wennberg J (1993). Geographic variation in physicians expenditures in the United States. NEJM; 328:621-7. 6. Fisher E, Wennberg J (2003). Healthcare quality, geographic variations, and the challenge of supply sensitive care. Perspectives in Biology and Medicine; 46(1):69-79. 7. U.S. Department of Health and Human services (2013). The Affordable Care Act (Website). Available: http://www.hhs.gov/healthcare/rights/law/index.html. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 160 8. Lean Six Sigma (2014). The Power of Lean Six Sigma (Website). Available Online: http://engineering.dartmouth.edu/sixsigma/. 9. Trustees of Dartmouth College (2014). Clinical Microsystems (Website). Available Online: http://www.clinicalmicrosystem.org/. 10. Northern New England Cardiovascular Network Disease Study Group (2014). Published Literature (Website). Available Online: http://www.nnecdsg.org/pub_lit_2.htm. 11. Godfrey MM, Oliver BJ (2014). Accelerating the Rate of Improvement in Cystic Fibrosis Care: Contributions and Insights of the Learning and Leadership Collaborative. BMJ Quality & Safety; 23:i23-i32 (doi: 10.1136/bmjqs-2014-002804). 12. Marshall BC, Nelson EC (2014). Accelerating implementation of biomedical research advances: Critical elements of a successful 10-year CF Foundation healthcare delivery improvement initiative. BMJ Quality & Safety Supplement. 13. Mogayzel P, Dunitz J, Marrow L, Hazle L (2014). Improving chronic care delivery and outcomes: The impact of the cystic fibrosis care center network. BMJ Quality & Safety Supplement. 14. Sabadosa KA, Batalden PB (2014). Individuals with cystic fibrosis, families and healthcare professionals: Co-producers of care and quality improvement. BMJ Quality & Safety Supplement. 15. Cheng EM, Crandel CJ, Beyer Jr. CT, et al. (2010). Quality Indicators for Multiple Sclerosis. Multiple Sclerosis; 16(8): 970 980. 16. American Academy of Neurology (August 2014). Multiple Sclerosis Quality Measures Set (Draft for Public Comment). 17. McDonald KM, Sundaram V, Bravata DM, et al. for the Agency of Healthcare Research and Quality (2007). Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Available: http://www.ncbi.nlm.nih.gov/books/NBK44008/figure/A25995/?report=objectonly. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 161 18. Nelson EC, Greenfield S, Hays RD, et al. (1995). Comparing outcomes and charges for patients with acute myocardial infarction in three community hospitals: An approach for assessing value. International Journal of Quality & Safety in Healthcare; 7(2):95-108. 19. Nelson EC, Batalden PB, Godfrey MM, Lazar J (2011). Value by Design: Developing Clinical Microsystems to Achieve Organizational Excellence. Josey-Bass: San Francisco. 20. Cretin S, Shortell SM, Keelar EB (2004). An evaluation of collaborative interventions to improve chronic illness care. Evaluation Review, 28(1), 28-51. 21. de Silva D (2014) for the Health Foundation Evidence Centre (2014). Improvement Collaboratives in Health Care.U.K. Health Foundation, London. 22. Kilo CM (1998). A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvements Breakthrough Series. Quality Management in Health Care, 6(4), 1-13. 23. Godfrey MM (2013). Improvement Capability at the Front Lines of Healthcare: Helping through leading and coaching (PhD Dissertation & Publications). Available Online: http://hj.diva-portal.org/smash/get/diva2:640804/FULLTEXT01.pdf 24. Godfrey MM (2013). Improvement Capability at the Front Lines of Healthcare: Helping through leading and coaching (PhD Dissertation & Publications). Available Online: http://hj.diva-portal.org/smash/get/diva2:640804/FULLTEXT01.pdf. 25. Godfrey MM, Andersson-Gare B, Nelson EC, Nilsson M, Ahlstrom G (2013, in Press). Coaching Interprofessional Health Care Improvement Teams: The Coachee, the Coach and Leader perspectives. Journal of Nursing Management. Doi: 10.1111/jonm.12068. [Epub ahead of print] 26. Grumback K, Bainbridge E, Bodenheimer T (2012). Facilitating improvement in primary care: The promise of practice coaching. The Commonwealth Fund, 1605(15), 1-14. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 162 27. Harvey G, Loftus-Hills A, Rycroft-Malone J, et al. (2002). Getting evidence into practice: The role and function of facilitation. Journal of Advanced Nursing, 37(6), 577-588. 28. Gustafson DH, Quanbeck AR, Robinson JM, et al. (2013). Which elements of improvement collaboratives are most effective? Addiction. DOI: 10.1111/add.12117 29. Rapkin BD, Weiss ES, Lounsbury DW, et al. (2012). Using the interactive systems framework to support a quality improvement approach to dissemination of evidence-based strategies to promote early detection of breast cancer: Planning a comprehensive dynamic trial. Am J Community Psychol. doi 10.1007/s10464-012-9518-6. 30. West SG, Duan N, Pequegnat W, et al. (2008). Alternatives to the Randomized Controlled Trial. Am J Public Health; 98(8): 1359-1366. 31. Huang X, Rosenthal MB (2014). Transforming Specialty Practice The Patient-Centered Medical Neighborhood. NEJM; 370(15): 1376-78. 32. Coleman K, Austin BT, Brach C, Wagner EH (2009). Evidence On The Chronic Care Model In The New Millennium. Health Affairs; 28(1): 75-85. 33. Sia C, Tonniges TF, Osterhus E, Taba S (2004). History of the Medical Home Concept. Pediatrics: 113:1473-78. 34. American College of Physicians (2010). The Patient-Centered Medical Home Neighbor: The Interface of the Patient-Centered Medical Home with Specialty/Subspecialty Practices. Philadelphia: American College of Physicians. 35. Institute for Healthcare Improvement (2014). The Plan-Do-Study-Act Cycle. Available Online: http://www.ihi.org/knowledge/Pages/Tools/PlanDoStudyActWorksheet.aspx. 36. Weinfurt, K. P. (2000). Repeated Measures Analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and Understanding Multivariate Statistics (pp. 317361). Washington, DC: American Psychological Association. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 163 37. Weissman NW, Allison JJ, Kiefe CI, et al. (1999). Available benchmarks of care: The ABCsTM of benchmarking. Journal of Evaluation in Clinical Practice;5:269-281. Available Online: http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=5185359&site=ehost-live&scope=site. 38. Benneyan JC, Lloyd RC, Plesk PE (2003). Statistical process control as a tool for research and healthcare improvement. Quality and Safety in Health Care;12: 458-464. Available Online:http://qshc.bmj.com/cgi/reprint/12/6/458. 39. Thor J, Lundberg J, Ask J, et al. (2007). Application of statistical process control in healthcare improvement: A systematic review. Quality and Safety in Health Care;16(5): 387-99. Available Online: http://qshc.bmj.com/cgi/reprint/16/5/387. 40. Johnson K, Brooks BR, Cohen JA, Ford CC, Goldstein J, et al. Copolymer 1 reduces relapse rate and improves disability in relapsing-remitting multiple sclerosis: Results of a phase III multi-center, double-blinded, placebo-controlled trial. Neurology. 1995; 45:1268-1276. 41. Panitch H, Goodin DS, Francis G, Chang P, Coyle PK, O'Connor P, et al. Randomized, comparative study of interferon beta1a treatment regimens in MS: The EVIDENCE trial. Neurology. 2002; 59(10):1496-1506. 42. Goodin D, Arnason B, Coyle P, et al. The use of mitoxantrone (novantrone) for the treatment of multiple sclerosis. Neurology. 2003; 61:1332-1338. 43. Kargiotis O, Paschali A, Messinis L, Papathanasopoulos P. Quality of life in multiple sclerosis: Effects of current treatment options. Int Rev Psychiatry. 2010; 22(1):67-82. 44. Trojano M, Pellegrini F, Paolicelli D, et al. Real-life impact of early interferon beta therapy in relapsing multiple sclerosis. Annals of Neurology. 2009; 66(4):513-520. 45. Bell C, Graham J, Earnshaw S, Oleen-Burkey M, Castelli-Haley J, Johnson K. Cost-effectiveness of four immunomodulatory therapies for relapsing-remitting multiple sclerosis: A Markov model based on long-term clinical data. Journal of Managed Care Pharmacy. 2007; 13(3):245-261. STATISTICAL METHODS FOR MULTIPLE SCLEROSIS 164 46. Barkhof F, Simon JH, Fazekas F, et al. (2012). MRI monitoring of immunomodulation in relapse-onset multiple trials. Nature Reviews Neurology; 8:13-21. 47. Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19(5, pt 2):496500.  ...

... Student Perceptions of Successful Performance in a Physical Therapist Assistant Program Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Michele R. Voight, PTA, MPA Copyright 12/08/2021 By: Michele R. Voight, PTA, MPA All rights reserved Approved by: Laura Santurri, PhD, MPH, CPH Committee Chair ______________________________ Stephanie P. Kelly, PT, PhD Committee Member ______________________________ Lisa Borrero, PhD Committee Member ______________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program Chair, Interprofessional Health & Aging Studies University of Indianapolis ______________________________ Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis ______________________________ STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 1 Student Perceptions of Successful Performance in a Physical Therapist Assistant Program Michele Voight Department of Interprofessional Health & Aging Studies, University of Indianapolis STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 2 Abstract Introduction: To develop retention strategies, it is necessary to understand students perceptions of successful performance and the factors that affect success. No literature has been found that addresses successful performance in a physical therapist assistant (PTA) program from the students point of view. Objective: This qualitative study examined PTA students perceptions of success in their PTA program and how that perception may have changed over time. The study investigated how internal factors such as self-efficacy and external factors such as family support and work are perceived by students to affect successful performance. Method: Eleven PTA students, who volunteered to participate, were selected from a large metropolitan community college. The purposeful sample was chosen to represent a variety of gender, race, age, family structure, and socioeconomic statuses. To address the primary investigator's bias, interviews were conducted by three research assistants who were in their second year of a Doctor of Physical Therapy (DPT) Program. Results: Four major themes emerged that create a picture of student perception of successful performance in a PTA Program: Measures of Success, Facilitators of Success, Barriers to Success, and Program Expectations. Facilitators and barriers to success themes had several overlapping sub-themes which included classmates, family, and mental state. Discussion and Conclusion: An understanding of the student perceptions will aid program administrators and faculty in the development of retention strategies which promote a sense of belonging and an atmosphere of positivity that can be beneficial for student persistence. Keywords: student performance, student success, persistence, retention, attrition, selfefficacy STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 3 Acknowledgments The completion of this study could not have been possible without the expertise and commitment of the dissertation committee from the University of Indianapolis, Chair - Dr. Laura Santurri, Analysis Expert Dr. Lisa Borrero, and Content Expert Dr. Stephanie Kelly. I am forever grateful for their feedback and guidance in this study and their dedication to education. I want to acknowledge the faculty of the University of Indianapolis Doctor of Health Science Program. Throughout this journey, they have provided knowledge and support which is the foundation of this research. I would also like to thank my research assistants who are students from Texas Womans University Doctor of Physical Therapy Program, Kaylee Hedricks, Emily Mowrey, and Breasha Blaylock. Their time and commitment to this research is much appreciated. In addition, I would like to thank Dr. Peggy Gleeson, Associate Director of the Texas Womans University Doctor of Physical Therapy Program for providing mentorship and guidance to the research assistants and for her expertise during the process of this study. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 4 Table of Contents Title Page .1 Abstract .2 Acknowledgments .....3 Chapter 1: Introduction .....6 Problem Statement .....7 Purpose Statement .7 Research Question .....8 Significance of Study .8 Definition of Terms .8 Chapter 2: Literature Review .9 Student Success ....10 Attrition, Retention, and Persistence ....12 Barriers and Facilitators of Success 15 Self-efficacy and Student Success 17 Success Interventions 19 Student Perceptions of Success Chapter 3: Method ....20 ....22 Study Design ....22 Participant ....23 Setting 24 Procedures ....24 Recruitment25 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE Informed Consent ...27 Data Collection Processes ...28 Data Management and Analysis Rigor/Trustworthiness Chapter 4: Results 5 ...29 ...32 ...33 Theme 1: Measures of Success ...34 Theme 2: Facilitators of Success ...36 Theme 3: Barriers to Success ...38 Theme 4: Program Expectations ...39 Chapter 5: Discussion & Conclusion ...41 Study Limitations ..49 Implications for Future Research ..51 Conclusion ..51 References ..53 Table 1: Participant Demographics ..63 Table 2: Participant Characteristics ..64 Figure 1: Visual Representation of Themes ..66 Appendices ..67 Appendix A: Study Recruitment Information Form ..67 Appendix B: Student Demographic Information Form .70 Appendix C: Email Notification of Participant Selection Appendix D: Consent to Participate in Research Appendix E: Interview Questions ..72 ..73 ..75 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 6 Student Perceptions of Successful Performance in a Physical Therapist Assistant Program Attrition in higher education is of great concern due to the costs to students, educators, institutions, governments, and taxpayers (Foo et al., 2018). Costs include resources such as time, money, effort, and opportunity but can also include the cost to self-worth (Foo et al., 2018; Linderman & Kolenovic, 2013). Studies on student success often focus on success predictors and the educational experience; however, a large percentage of student attrition continues to be unexplained (Bowman et al., 2019). Although institutions develop retention programs to help students succeed, few are built on empirical assessment (Kebaetse et al., 2018). Specifically, there is a gap in the literature regarding the student perspective on how success is defined. This lack of information may impede the success of such retention programs. Community colleges often face additional challenges regarding student retention due to poorly defined degree pathways, campus culture, low academic preparation of the student, and competing external factors and responsibilities for the student (Linderman & Kolenovic, 2013). Gregor et al. (2020) noted that the graduation rate within six years of matriculation is 57% for full-time community college students and 39% for part-time students. Competitive admission health care programs, such as physical therapist assistant (PTA) programs, are often located in community colleges and struggle with student attrition. For PTA programs to be viable, they are required to meet the benchmarks of their institutions, as well as the requirements of a third-party accreditor. Although institutional benchmarks may differ between colleges, all PTA programs are required to be accredited nationally through the Commission on Accreditation in Physical Therapy Education (CAPTE) (American Physical Therapy Association, n.d.). CAPTE requires a two-year graduation rate of 60 percent for all PTA Programs (CAPTE, 2016). This standard is STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 7 higher than the statistics noted by Gregor et al. (2020). Because PTA programs must prioritize accreditation standards, there is a strong focus on retention and student success. Typically, admission into a PTA program is based on specific predictors of academic success, such as Grade Point Average (GPA) and standardized testing. However, these programs may continue to experience a high rate of attrition and poor student performance (Kabiri et al., 2017). According to Kebaetse et al. (2018), most students who enter a health care education program have the intellect to succeed as reflected by admission criteria; however, many other external factors play a significant role in determining a students successful performance and goal achievement. Problem Statement Academic success, as addressed in the literature, is typically measured in grades, retention, and completion outcomes, which meet the needs of the academic institution. However, the student perspective or definition of successful performance often is overlooked. In recent years, research has examined student self-efficacy and resilience interventions and the effects on student success. Specifically, researchers have found a positive correlation between academic performance and both self-efficacy and resilience in two Doctor of Physical Therapy (DPT) programs. (Mejia-Downs, 2020; Stickley et al., 2019), though no such correlation has been established for students in PTA programs. Some existing studies describe certain aspects of academic success in a PTA program, such as admission criteria (Kabiri et al., 2017); however, no studies appear to exist that explore and identify PTA student perceptions of success. To address attrition and poor performance, educators must recognize and appreciate factors that influence success from the student perspective. Purpose Statement STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 8 The purpose of this basic interpretive qualitative study was to explore student perceptions regarding factors that influence performance in a PTA program. Research Questions How do students perceive successful performance in a physical therapist assistant program? o How do physical therapist assistant students perceive the impact of external factors on their performance/success? o How do physical therapist assistant students perceive the influence of selfefficacy on their performance/success? o How does physical therapist assistant students perception of success change throughout the program? Significance of the Study The potential impact of this qualitative study is to provide a deeper and richer understanding of the student perspective. The results serve to inform the development of strategies and processes that directly address the factors that influence PTA students in their attempts to be successful in the program. Definitions of Terms Physical therapist assistant (PTA) education Prepares graduates with knowledge and skills to enter the profession. PTAs graduate from a CAPTE-accredited, associate degree PTA program and must pass the National Physical Therapy Examination (APTA, n.d). Self-efficacy A persons belief about their capability to complete a task (Bandura, 1994). STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 9 Student attrition the percentage of students who leave a program before completion (Foo et al., 2018). Attrition outcome measure calculated as the percentage of matriculated students leaving a program before completion (Beer & Lawson, 2017; Foo et al., 2018). Retention retaining students from semester to semester or year to year during their academic career (York, 2015). Persistence - degree completion (York, 2015). Literature Review Community college graduates comprise a substantial percentage of health care professionals (Staykova, 2012). These institutions and health care programs have additional challenges that four-year institutions do not face (Staykova, 2012). The community college student population is diverse and may encounter external factors that may inhibit their success (Staykova, 2012). In community college allied health care programs such as physical therapist assistant, dental hygiene, and radiography, the number of applicants can outnumber the program's available seats (Easley, 2016; Ingrassia, 2016). Therefore, it is imperative to have reliable admission criteria. It is equally essential to provide interventions throughout the program to promote student success (Denham et al., 2018; McComb & Lyddon, 2016). According to Mitchell et al. (2021), strong admission requirements cannot stand alone, and programs must also include quality retention strategies. In programs with a clinical application aspect included, such as health science programs, academic success does not always translate to clinical application success, making admission criteria alone challenging to predict success. When a student is not academically successful, a seat in the program remains empty until the following year, which is a financial burden for the institution (Ingrassia, 2016). The student STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 10 also bears the weight of the burden due to educational delay, postponement in possible earnings, and the possibility of adverse psychological effects from failure (Ramirez et al., 2018). Addressing student success from the health care programs perspective provides only a partial view of the problem with student retention. Understanding the students perceptions of successful performance and the barriers they face can help educators in allied health programs facilitate evidence-based success interventions (Denham et al., 2018). Student Success The terms student academic success, student success, and successful student performance in higher education are often interchangeable in the literature (York et al., 2015). Defining academic success or successful performance is difficult because the perception of that definition will differ depending on the organization or the individual student (Denham et al., 2018; York et al., 2015). Also, the higher education student population has changed in recent years. The non-traditional student is now the majority; therefore, the definition of student success and success outcomes may need to be reconsidered (Branon, 2018). Although the definition of retention, student success, and methodologies for measuring the outcomes differ among higher education institutions, most agree there is a need to address and prioritize student success (Dwyer, 2017). Student success, especially in allied health programs, notably physical therapist assistant programs, has traditionally been defined in a variety of ways, such as persistence, retention, ontime graduation, program completion, grade point average (GPA), and board examination pass rate (Kabiri et al., 2017; Maring et al., 2013; Wheeler et al., 2018; York et al., 2015). However, these definitions may not always align with students perceptions of success (Denham et al., 2018). Bowman et al. (2019) argue that non-cognitive factors such as self-efficacy and grit may STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 11 also affect persistence and graduation rate above their relationship with GPA. A study by Leiken (2017) found that dental hygiene students in a two-year program were more successful if they already had a higher education degree, were engaged in social activities on campus, and participated in active learning activities. York et al. (2015) conducted a literature review to more accurately define academic success, resulting in the following definition, inclusive of academic achievement, attainment of learning objectives, acquisition of desired skills and competencies, satisfaction, persistence, and post-college performance (p. 5). York et al. (2015) further acknowledged that using academic achievement alone to measure student success does not address individual cognitive abilities and may not be a valid measure due to grading methodology inconsistencies. Educational programs including those in various health care disciplines, often conduct research on student success to ensure that admitted students have the best chance of a successful outcome (Kabiri et al., 2017; York, 2015). According to Easley (2016), many students in an allied health program in a community college are at high risk for failure due to their nontraditional and low economic status and college readiness. Success in physical therapy programs is assessed using various measurements that include cognitive and non-cognitive factors; however, research on the student perception of how these factors affect success is limited (Cherry and Blackinton, 2017; Cassidy et al., 2020). An abundance of published studies using retrospective data for predictive models for student success exists in the literature for students in nursing programs, medical schools, and physical therapist programs. A review of the Journal of Physical Therapy Education (JOPTE) revealed that four studies on student success have been published regarding the prediction of physical therapist assistant (PTA) graduation rate and licensure pass rate. Each of these studies was retrospective and examined variables such as STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 12 admission criteria, length of the program, lab experiences, faculty characteristics, the length of clinical education experiences, accreditation status, and a comparison of for-profit and not-forprofit institutions (Demarais et al., 2011; Gresham et al., 2015; Kabiri et al., 2017; Maring et al., 2013). For example, one study found that the GPA of PTA students was the best predictor of licensure pass rate (Demarais et al., 2011). Another determined that overall prerequisite GPA and standardized test scores were predictors of graduation and licensure pass rates (Kabiri et al., 2017). Maring et al. (2013) found that accreditation compliance and meaningful lab experiences were positively correlated with success on the licensure exam. Gresham et al. (2015) found a significant positive correlation between PTA programs with a greater percentage of weeks with full-time clinicals and graduation and licensure pass rates. Although studies of PTA and other health care programs exist using quantifiable retrospective measures, attrition continues to be a concern, and solutions to improve student success appear to be elusive. Attrition, Retention, and Persistence Student attrition is calculated as the percentage of matriculated students leaving a program before completion and is challenging to address due to the fluidity of the problem (Beer & Lawson, 2017; Foo et al., 2018). Retention and persistence are used to measure academic outcomes. Retention is defined as retaining students from semester to semester or year to year during their academic career, whereas persistence is defined as degree completion (York et al., 2015). When a student fails to complete a program, there is a negative effect on the student, the institution, the workplace, and society (Cassidy et al., 2020). In the United States, community colleges enroll half of all undergraduate students, yet in 2012, these institutions had a completion rate of 22% (Linderman & Kolenovic, 2013). Gregor et al. (2020) explained that community colleges tend to enroll a high percentage of students from STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 13 marginalized groups in terms of socioeconomic status, race/ethnicity, age, and generational status. These students may encounter many barriers while attending college, such as financial constraints, work obligations, familial responsibilities, and lack of academic preparation, which may affect their ability to succeed (Gregor et al., 2020; Linderman & Kolenovic, 2013). According to Gregor et al. (2020), the community college student population is underrepresented in the literature. There is minimal investigation of their perception of what barriers they face and what could enable their success. In response to the underserved populations barriers, President Barack Obama enacted America's College Promise (ACP) to support access to higher education and economic development (Palmadessa, 2017). The ACP was directed at community colleges to increase the number of college graduates and educated workers by offering improved access to higher education to underserved populations through free tuition for two years for qualifying students (Palmadessa, 2017, p. 13). Although this legislation may have had a notable impact on the economy and the public, it may have added additional burdens on community colleges (Palmadessa, 2017). Specifically, this policy may have led to the admission of academically unprepared students and the ensuing financial burden to provide additional resources, such as supplemental learning, to meet these students' needs to promote their success (Palmadessa, 2017). In addition to the ACP initiation, the 2015 Complete College America (CCA) agenda called for community colleges to increase degree completion by 50% (Easley, 2016). Unfortunately, noble legislation is only as good as the current political support. If funding is not made available to support the ACP and CCA, community colleges will continue to struggle to improve outcomes and meet the needs of the public (Easley, 2016; Palmadessa, 2017). STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 14 Due to legislative agendas, community colleges face a tremendous challenge due to outcome-based funding and continuing high attrition (Easley, 2016). According to the National Center for Education Statistics (2020), of the 52% of undergraduates enrolled in subbaccalaureate programs, health science programs were the most common field of study. A majority of these programs were located in public two-year institutions with a large proportion of female enrollment, over 25 years old, and black or Hispanic (National Center for Education Statistics, 2020). As mentioned, individuals in these groups typically face additional burdens due to family and work obligations and ethnic minority challenges coming from a historically underserved population (Gregor et al., 2020). Of the 416 accredited and developing PTA programs in the United States, well over 50% are located in community colleges with a wide range of student demographics depending on the program's location (CAPTE, 2020). The mean graduation rate in 2019 for all accredited PTA programs was 85.6%, and the mean first-time licensure pass rate in 2019 was 84.2% (CAPTE, 2020). Although this statistic meets accreditation standards, over 15% of PTA students are not successful, which results in institutional and student burden. For community colleges, specifically PTA programs, to meet the legislative initiatives and required outcomes, a need is present to develop strategies to address these barriers and promote student success (Easley, 2016). A certain amount of attrition in health care programs is expected due to high levels of skill competency required for public safety and practice standards. However, attrition impacts the workforce, with fewer successful graduates to meet the high demand for health care professionals (Cassidy et al., 2020). Wray et al. (2014) found that almost 50% of United Kingdom university nursing students surveyed considered dropping out of their program at some stage in their educational journey. Although existing literature examines attrition among nursing STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 15 and medical students, there is a need to explore further and understand why some students in two-year sub-baccalaureate health care programs, notably PTA programs, succeed when others fail (Cassidy et al., 2020; Kabiri et al., 2017; Kebaetse et al., 2018). Barriers and Facilitators of Success Minimal published research has examined the characteristics and perceptions of those students who succeed. Research has mainly focused on barriers to success that students may face and identifying those at greater risk (Wray et al., 2014). Community colleges have a high number of students enrolled who are low-income, ethnic minorities, and first-generation students. As noted above, students in these groups often face numerous barriers to academic achievement and successful performance, yet many do succeed (McFadden, 2016). According to Gregor et al. (2020), students from some racial and ethnic backgrounds may experience barriers due to the expectation that they prioritize family needs over individual aspirations (Gregor et al., 2020). Language can also serve as a barrier, as nursing students for whom English is a second language tend to have a higher attrition rate than other students (Denham et al., 2018). Accordingly, students in health care programs for whom English is not their first language may experience communication challenges during clinical experiences (Wray et al., 2014). In previous literature, Utzman et al. (2007) found student ethnicity to be a significant predictor of academic difficulty in physical therapist education. In contrast, the prediction study cited above on PTA student graduation and licensure success by Kabiri et al. (2017) did not find that ethnicity contributed to their prediction model. In addition to personal characteristics such as background, students often face other barriers to success, including increased stress and anxiety. According to Gautman et al. (2020), high anxiety, stress, and depression are prevalent problems among college students. Gautman et STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 16 al. (2020) explains that the transitional period in college students lives, the increased academic demands, and balancing a social life places additional strain on the student. Gautam et al. (2020) and Mejia-Downs (2020) agree that health profession students, including physical therapist students, experience an increased stress level compared to other college students of the same age. In addition, they noted that the stress from academic demands was a more significant barrier to success than personal characteristics or financial issues. Additional stress may be added to the student in a two-year health care program due to the curriculum's fast pace, expanding role of professional practice needs, and the high expectation of proving skill competency (Ingrassia, 2016; Staykova, 2012). Gautam et al. (2020) further emphasized that the effects of collegerelated stress and anxiety may extend beyond the college experience and may lead to future health issues and disabilities. According to Denham et al. (2018), barriers to success must be addressed, while strategies to facilitate success must be implemented. In their systematic review, Schneider and Preckel (2017) concluded that social interaction and meaningful learning experiences are positively associated with student achievement. However, the authors acknowledge that varying characteristics of the learning experience and the environment make identifying specific facilitators of success challenging. A qualitative study by Wray et al. (2014) found that facilitators of degree completion included healthy family and peer support, clear educational expectations, and a strong desire to become a health care practitioner. Conversely, Hepworth et al. (2018) suggested that the perception of institutional commitment to student success and student social integration into college life were considered secondary factors and not found to be a significant predictor. Facilitating academic preparation was found to be significant in predicting success (Hepworth, 2018). STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 17 Self-Efficacy and Student Success The college experience can be a new adventure into autonomy and result in personal growth. However, it can also be a transition time that results in high anxiety and unhealthy stress, lowering students belief in their academic and performance abilities (Ramirez et al., 2018). Due to the lack of consensus for an optimal admission process based on cognitive factors that lead to success in DPT programs, non-cognitive factors such as self-efficacy, emotional intelligence, and resilience should be explored to improve student success (Van Veld, 2018). Bowman et al. (2019) agree and found a direct positive relationship between non-cognitive attributes and college GPA. Self-efficacy is defined by Albert Bandura as a persons belief about their capability to perform a specific task at a certain level (Bandura, 1994). Van Veld et al. (2018) defines self-efficacy as the confidence in DPT students ability to use coping strategies to make a positive change in a stressful situation. When reviewing the relationship between self-efficacy and self-confidence, Blanco et al. (2020) stated that self-efficacy is the interaction between a person and a specific task whereas self-confidence is a personal characteristic. Further, Bandura (1997) defines self-efficacy as a construct within a theoretical system and confidence as a general term without specificity. Blanco et al. (2020) found that undergraduate students who are more self-confident despite facing challenges, such as those experienced during the pandemic, are likely to possess higher self-efficacy compared with students who exhibit less selfconfidence. Students belief in their capabilities, whether it be to perform a specific task or manifested as a general ability to succeed, often determines their ability to succeed (Ouweneel et al., 2013). Many students can persist academically and achieve positive outcomes by completing their programs, yet others fail to complete; this may be due to their actual ability to perform or their overall belief in their ability (Ouweneel et al., 2013). STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 18 Students who fail may experience low self-efficacy (Ramirez et al., 2018). Students with high self-efficacy have been observed to look at difficult tasks as a challenge and do not approach challenging tasks as a threat to their success (Bandura 1994; Rowbotham & Schmitz, 2013). According to Gregor et al. (2020), students are more likely to overcome barriers to achieving their goals if they believe they have the capability of succeeding. Stickley et al. (2019) further suggested that people with high self-efficacy are more willing to attempt a task and demonstrate more remarkable persistence to complete the task with a successful outcome. A study conducted by Ouweneel et al. (2013) claimed that in an experimental controlled setting, as the level of self-efficacy changed over time, so did study engagement and task performance. As self-efficacy increased, the students study engagement and task performance improved (Ouweneel et al., 2013). Self-efficacy depends on the activity of influence, such as a specific test performance; therefore, it cannot be assumed that the level of self-efficacy will remain stable over time (Ouweneel et al., 2013). Van Veld et al. (2018) found that first-year DPT students gained confidence over time in their ability to cope with stress and skill performance. In addition, instructor affirmation interventions can improve the student and teacher relationship, which can be an important driver of success (Smith et al., 2021). If faculty invest consistently in increasing students self-efficacy through positive reinforcement and short-term goal setting, it may improve student success (Ouweneel et al., 2013). Similar to the need for high self-efficacy in DPT programs, high self-efficacy may be equally crucial in community college workforce programs such as PTA programs due to the same rigorous expectation of applying learned skills in testing and clinical environments (Easley, 2016; Kabiri, et al., 2017). The academic rigor of a health profession program can be overwhelming for many students. Identifying their level of self-efficacy during the program may STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 19 help faculty implement strategies to develop and build self-efficacy to improve successful performance (Rowbotham & Schmitz, 2013; Van Veld et al., 2018; Stickley et al., 2019). Success Interventions Interventions are often developed in institutions and programs to improve student success; however, it is common to create such programs without evidence-based program evaluation or fully understanding all aspects involved in program initiatives (McComb & Lyddon, 2016). According to McComb and Lyddon (2016), colleges usually look at lagging indicators such as graduation rates retrospectively without considering leading indicators, such as participation in study groups, class attendance, and task performance. Although lagging indicators, such as completion rates, are essential to assess, the view looks at the past and what has already occurred. Assessing leading indicators for the development of interventions informs educators of what is happening in the present. In addition, leading indicators can serve as a predictor of results likely to occur. Using the formative assessment of leading indicators allows for corrections and adjustments of interventions to positively impact student success (McComb & Lyddon, 2016). Linderman and Kolenovic (2013) promote the provision of an inclusive environment to accentuate a sense of belonging and a sense of ownership of a specific career. In a qualitative study, Cassidy et al. (2020) found that a positive environment, a sense of belonging, and connection with faculty enhanced success in physiotherapy students. An appreciative inquiry research approach performed in the United Kingdom by Masika and Jones (2016) found similar results of the importance of an inclusive environment. Jacobs et al. (2015) found that a peer mentorship program was beneficial in a two-year nursing program due to the high rigor of health education. Peer-to-peer interaction provided emotional support and educational benefit for both STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 20 the mentor and the mentee (Jacobs et al., 2015). The findings demonstrated that the development of an educational community of practice provides sustained engagement between students and faculty and peer to peer, which creates a sense of belonging (Masika & Jones, 2016). In addition to addressing academic needs in college students, it is vital to promote interventions that attend to this population's mental health and well-being due to adolescence's transitional period to adulthood (Deasy et al., 2014). Bogardus et al. (2021) found increased reported stress and anxiety in first-year DPT students and suggests that physical therapy programs may want to implement strategies to manage factors affecting mental health. TraskKerr (2019) suggested using positive psychology interventions (PPI), which focus on nurturing the students' well-being versus only focusing on academic success. The use of PPI focuses on social harmony and builds hope and confidence, which has been found to have a positive correlation with success (Trask-Kerr et al., 2019). Many of today's college students are of the millennial generation. They have been described as being catered to, feeling entitled, having a fear of failure, and being hyper-connected with the world of technology (Ramirez et al., 2018). This generation may benefit from positive psychology as faculty can re-frame failure as a quality learning experience and an opportunity for growth (Ramirez et al., 2018; Trask-Kerr et al., 2019). Faculty are in an excellent position to develop and participate in interventions to address student needs; however, speculating what those needs are without receiving student input results in an ineffective product (Denham et al., 2018). Student Perception of Success Although student success has been studied at length, student perceptions of their own success is limited in the literature. York et al. (2015) noted that their definition of academic success excluded the student's perspective and was a limitation of their review. A qualitative STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 21 study by Abdulghani et al. (2014) found that undergraduate medical students in Saudi Arabia perceived program success depended on academic strategies, managing external problems, using available resources, and self-motivation. The authors stated that an in-depth analysis of student perception is needed to truly understand the factors that lead to high achieving health care students success. Goldingay et al. (2014) found that first-year social work students perceived those personal qualities such as determination, work/life balance, and confidence were critical in facilitating success as academic skills. In a qualitative exploration of United Kingdom student physiotherapists perceptions of success, Cassidy et al. (2020) described looking at student success in the sociocultural realm as a burgeoning concept. The authors found that students' perception of why they achieved success was due to a sense of belonging, positive interactions with faculty and peers, and a healthy self-identity (Cassidy et al., 2020). Looking at students' sociocultural factors in an educational program may decrease marginalization and promote a more culturally aligned environment, resulting in greater student success (Cassidy et al., 2020). Denham et al. (2018) reported that unless faculty consider student perceptions of success when developing retention interventions, there is a likelihood that outcomes of these interventions will not meet expectations. In addition, faculty need to analyze their perceptions of student failure while better understanding their students' perceptions of success (Ramirez et al., 2018). Many external variables may play a role in a student's ability to succeed. Therefore, there is a need for further investigation to better understand how identified external factors influence student success (Cherry & Blackinton, 2017; Denham et al., 2018). As community college health care programs struggle with retention, they continue to be relied upon as a significant contributor to the health care workforce (Staykova, 2012). The diverse population of community college health care programs, high academic rigor, clinical STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 22 competency expectations, and the unique external factors students face make success a continuous challenge (Ingrassia, 2016; Leiken, 2017). Research on student retention from the institutional perspective has been conducted in two-year, four-year, and graduate school health care programs, yet attrition continues to be a problem. A different research approach is needed to meet the societal demand of the health care workforce and relieve the burden of costs to the student and educational institution. Further investigating the student perception of successful performance will help institutions and educators meet the student's needs. Understanding student attrition and successful performance are unlikely to be achieved unless the student perspective is considered. It is, therefore, essential to understand the student's view of success in an academic environment. If this understanding is not achieved, it is impossible to align realistic institutional academic expectations with success outcomes (Denham, 2017). To date, no published reports on the PTA student's perception of success have been found. Understanding student perceptions of success will help PTA programs and faculty develop effective resources and strategies to support the students' needs and meet institutional and program required and desired outcomes. Method Study Design This study utilized a qualitative basic interpretive approach. This approach focused on how people perceive and interpret their experiences and the meaning assigned to that interpretation (Merriam, 2002). In this study, the researcher explored the perceptions of successful performance among physical therapist assistant (PTA) students enrolled at a community college and attempted to understand if this perception changed over time during the PTA program. The researcher also explored students' perceptions of how external factors and STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 23 self-efficacy affected their performance. Institutional Review Board approval was granted prior to the commencement of any study activities. Participants Purposeful sampling was used to recruit participants who represented varying demographics in the Houston Community College (HCC) PTA Program. Purposeful sampling is an intentional strategy to include participants who will best inform the researcher of the phenomenon being investigated (Creswell & Poth, 2018). This approach was appropriate for the study because the researcher was able to select each participant to ensure they had experienced the phenomenon under study and met the inclusion criteria (Creswell & Poth, 2018). To be included in the study, an individual had to be a current HCC PTA Program student enrolled full time (at any level in the program) who had successfully earned a passing grade in all required courses in the first semester in the program. There were no exclusion criteria. Approximately 85 PTA students were informed of the study and opportunity for participation. Determining an adequate sample size for a robust qualitative study can be challenging. Malterud et al. (2015) recommend the use of information power represented in a five-item model. The model consists of the study aim, sample specificity, use of a theory, quality of dialogue, and analysis strategy. According to Creswell and Poth (2018), generalization is not the goal of qualitative research; rather, the goal is to provide insight into a population's specific experience. This study included 11 participants representing a range of the various student characteristics and demographics found in the HCC PTA Program. Although the study was not grounded in an established theory, the study aim, and specificity were narrow. The study focused on a perception of success and performance experiences from one specific PTA program. The quality of dialog during the interviews was strong because the interviewers had relatable STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 24 experiences as students themselves and were mentored in qualitative interviewing. The analysis was an in-depth exploration of the phenomenon of study from deliberately chosen participants that represented various student characteristics of the HCC PTA Program. Using the rationale from Creswell and Poth (2018) and Malterud et al. (2015), the sample size of 11 participants was adequate. Setting PTA program students were recruited from HCC, a large urban metropolitan community college located in Houston, Texas. The HCC PTA Program student demographics represent the HCC student body, which has a large international and minority population. The median age of the student population is 27 but ranges from 1760 years old. Many of the students are raising a family, working, and are first-generation college students (Houston Community College, n.d. About section). HCC has 14 Centers of Excellence and over 20 campuses. The PTA program is accredited by the Commission on Accreditation in Physical Therapy Education and is located in the Health Science Center of Excellence on the John B. Coleman Campus. The campus is located in the Texas Medical Center, which is the largest medical center in the world (Houston Community College, n.d. About section). The HCC PTA Program is a five-semester program with 66 credits and culminates in an Associate of Applied Science Degree with eligibility to sit for the National Physical Therapy Examination. The PTA program has a selective, competitive admission process. Annually the program has approximately 145 applicants for 52 available seats (Houston Community College n.d. Physical Therapist Assistant Program section). Procedures Recruitment STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 25 Recruitment took place at the beginning of the 2021 spring semester. Given this timing, PTA students were in the second or fifth and final semester of the program. The primary researcher, the PTA program director, introduced the research study to the first-year cohort during each of the six sections in a face-to-face laboratory class meeting of PTHA 2301 Data Collections. The second-year cohort was introduced to the study in the two sections of the faceto-face laboratory class meeting of PTHA 2339 Professional Issues. All enrolled PTA students and each course lead instructor were present during the study's introduction and discussion. The lead instructors did not participate in the introduction of the study discussion. The study and opportunity to participate was explained through a verbal discussion and a hard copy of the study recruitment information form (see Appendix A) stating the study purpose, potential participants sought, data collection, and data management processes. During the discussion, the researcher/PTA program director emphasized that participation in the study had no bearing on program status or grades. Students were informed that unique identification numbers and pseudonyms instead of legal given names would be used to identify those who were willing to participate, and those participants selected, respectively. Students were also informed that all contact and the actual interviews would be completed by the research assistants and not the program director. Students were informed that the program director would transcribe the interviews, and that, although pseudonyms were used during the interview, there would be the possibility to determine the identity of the participant during the analysis of the transcript. The study recruitment information form provided to the students explained the study's purpose and the opportunity to participate. In addition to the study recruitment information form describing the study, the students also received a student demographic information form (see Appendix B) that required the prospective participant to provide contact information and STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 26 demographic information. The first question on the student demographic information form asked if the student was willing to participate. The student demographic information form also included age, gender, current semester in the program, ethnicity, marital/partnership status, number of children, employment status, student debt, and housing status. Each student demographic information form included a unique identification number at the top of the page for participant identification purposes. Contact information collected consisted of the prospective participant's cell phone number and preferred email address. The contact information was used for communication purposes by the interviewers and was not used by the primary researcher. After the study was introduced, the program director left the room and was not present when students expressed interest to avoid pressure to participate. The students and the lead instructors remained until the end of class, at which time the lead instructors reminded the students of the opportunity to participate. All students were asked to submit the student demographic information form in a designated envelope prior to leaving the classroom, regardless of their choice to participate. This process was used to avoid being identified by others in the room as a possible participant. It was explained that this timeframe was the only opportunity provided for the students to express interest in participating in the study. The envelope was secured and only in possession of the lead faculty member of the two courses where the study was introduced and given directly to the research assistants who were present at the end of each class. The research assistants entered the participant information into a passwordprotected Excel spreadsheet developed by the primary researcher. The research assistants identified the prospective participant using the study identification number on the top of the demographic information form. The completed Excel spreadsheet was emailed by the research assistants to the primary researcher. The primary researcher used the spreadsheet to select 11 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 27 participants representing varying characteristics proportional to those commonly found in the HCC PTA Program. The completed paper demographic information forms were returned to the original envelope and placed in a locked cabinet by one of the research assistants. The envelope remained in the research assistant's possession until given to the primary researcher at the end of the study. All demographic information forms were destroyed at the completion of the study. Within one week of receiving the Excel spreadsheet of the students willing to participate, the primary researcher sent the list of selected participants to the research assistants. The research assistants notified the participants, using their provided email address (see Appendix C), that they were selected to participate in the study. The email explained to the participants that they should expect a phone call from a research assistant to schedule an interview. The research assistants contacted the participants two to four weeks later to schedule the interview. Informed Consent Once the selection process took place, a research assistant contacted the participants by phone to schedule the interview. During the phone conversation, the participants were provided an opportunity to have all of their questions answered, and received a reminder about the study's purpose, benefits, the research/method process, and the time commitment required. Participants were informed that all data collected would be de-identified and remain confidential outside the study's bounds. Before the phone call concluded, the participants were informed of their right to terminate participation anytime during the study, and participants provided verbal consent to schedule an interview. Participants were offered an opportunity to ask questions before the start of the interview. Written consent (see Appendix D) from the participant was emailed to the research assistants conducting the interview prior to the start of the interview. Verbal consent STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 28 was then given at the beginning of each interview for participation and audio-recording of the interview. Data Collection Process Data collection consisted of a virtual interview with each individual participant. To avoid bias from the researcher or student fear of retaliation due to the relationship between the primary investigator and the participants, three research assistants unaffiliated with the HCC PTA program performed the interviews. The research assistants (RAs) were second-year Doctor of Physical Therapy (DPT) students from Texas Womans University (TWU). The RAs were not experienced in qualitative research methods. As part of their DPT independent study research course requirement, instruction and mentoring were provided by the studys primary investigator and a TWU DPT faculty member, who is an experienced qualitative researcher and the RAs research advisor. Mentoring included each RA's requirement to read at least four articles regarding the process of qualitative research, followed by a face-to-face discussion about qualitative methodology. The RAs were also provided additional information on informed consent, confidentiality, sampling, interviewing, and data analysis. Each RA performed two practice interviews using the study interview guide with the primary investigator and TWU advisor. The primary researcher developed interview questions (see Appendix E) with input from the RAs and the TWU advisor. Each interview was conducted using the same format. All interviews were conducted through video conferencing using Zoom due to COVID-19 restrictions and the need to practice social distancing per guidelines from the Centers for Disease Control and Prevention. Each interview was conducted by two RAs and was audio recorded. Interviews began with an introduction of the RAs and a reminder of the purpose of the study. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 29 Participants were asked to provide a pseudonym for identification purposes that was used during the interview and data analysis. Although written consent was received prior to the start of the interview, verbal consent to participate was re-established, and permission to audio record was received. The participants were informed that the questions would start broad then move to more focused questions to ensure that a thorough picture of their perceptions was collected. Participants were informed that each interview was expected to last approximately 45-60 minutes and guided by a predetermined open-ended list of questions in a semi-structured format. Probing questions as a step described by Turner (2010) were used to follow up and clarify participants answers or allow for added breadth and depth to the participants comments. After all questions were addressed, the interview was concluded by asking the participants for additional comments. After each interview, participants were informed that a follow-up contact by email or phone might be necessary if additional information was gleaned from subsequent interviews during the data collection period. They were also informed that after data analysis, they would receive a copy by email of the conclusions and generated themes to ensure accuracy. Data Management and Analysis The primary investigator downloaded the audio recordings of the interviews to Temi, an advanced speech recognition software. Temi uses TLS 1.2 data encryption and secure servers to protect data (Temi, 2020). Transcriptions were downloaded to Dedoose (version 8.3.35), a computer-assisted qualitative data analysis software (CAQDAS), which was only accessible by the primary researcher. Transcriptions were also available to the research team through a private Google Drive. Audio recordings were deleted from the recording device after transcription to maintain the data's security and confidentiality. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 30 Following transcription, the primary researcher analyzed the data in collaboration with the RAs. The research team used a field journal in Dedoose (version 8.3.35) and Excel from the study conception through completion. As part of the field journal, memoing was used to establish an audit trail throughout data collection and analysis. Memoing provided a method for reflexivity and was used to maintain consistency among the research team (Birks et al., 2008). Memoing was used as part of the inductive process to document patterns identified from the data and determine if there were expected concepts not being seen in the data (Satterlund, n.d.). Memoing was also used in the raw data analysis to open communication and explore similarities and differences among the research team to collaborate on code and statement definitions (Birks et al., 2008). As perspectives changed throughout the research process, the field journal justified when and why the change occurred (Birks et al., 2008). Each member of the research team used Excel to individually perform a first cycle analysis of each transcription using line-by-line coding to identify words and phrases that emerged and were related to the research questions and the overall storyline (Stuckey, 2015). Codes were then assigned to the identified phrases and words. The first cycle resulted in each research team member identifying and documenting conceptual groupings of the codes that reflected commonalities and relationship clusters as described by Marshall and Rossman, 2011. A "lean coding" approach was used in the coding process to ensure only essential relevant codes existed (Creswell & Poth, 2018, p. 190). As suggested by Creswell & Poth (2018), this approach was used by starting the code development small with each research team member, then expanding as the code cycle collaboration progressed. The process was followed by a final review to again reduce the codes to a manageable number for theme development. The primary investigator entered the identified codes into Dedoose (version 8.3.35) and tagged related transcript excerpts. Additionally, a STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 31 codebook was developed in Excel to represent the codes' scheme to assist with organization and interpretation (Saldana, 2014; Stuckey, 2015). The codebook consisted of identified codes, code definitions, and where the codes were found. For member checking, the study participants were contacted by email by the RAs and sent an abbreviated summary of their interview. This process helped ensure the credibility of data collected by asking participants for adjustments or corrections to the researchers understanding of the transcription. The primary investigator collaborated with the RAs for a second cycle coding analysis to check for the need to reconfigure codes, as needed, and add or remove text to existing codes (Saldana, 2014). Following the second cycle of coding, the research team met virtually through Zoom to verify the accuracy of the codes. This process was used to add credibility to the study by triangulating the analysis's outcome within the research team (Carter et al., 2014; Henderson & Rheault, 2004). A thematic analysis was conducted by the primary researcher to draw out concepts and themes from the coding outcome (Saldana, 2014). The analysis determined the emergence of major themes (Saldana, 2014). Once themes were identified, a visual representation of PTA students' perceptions of successful performance was developed by the primary researcher. This image illustrated connections and synthesized key concepts discovered through data analysis (Verdinelli & Scagnoli, 2013). To ensure understanding of the phenomena under study, the research team performed investigator triangulation to evaluate the developed themes and visual display to secure intercoder reliability. This method added breadth to the study and fostered the credibility of the final data outcome (Carter et al., 2014). Triangulation was performed by each member of the research team reviewing the data and the visual representation in Dedoose (version 8.3.35) and Excel. As a second member checking method to ensure accuracy, STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 32 participants were sent a copy of the final thematic analysis by email with an opportunity to provide comments. An external audit was performed by the RAs qualitative TWU DPT advisor to reduce researcher bias. The advisor reviewed the data collected to determine if her conclusions were comparable to those of the research team. The advisor provided comments to the research team following the audit. All comments from the participants, the external auditor, and the research team were discussed via a virtual meeting using Zoom following the review. Adjustments to the final data outcome were completed as needed based on feedback from the participants and the research team. The analysis was complete following the final adjustments. Rigor/Trustworthiness Gubas Model of Trustworthiness was used to ensure rigor and address the reliability and validity of the data collected and analyzed (Henderson & Rheault, 2004). The model contains the four assessment components: credibility, transferability, dependability, and confirmability (Henderson & Rheault, 2004). Each component was addressed in the proposed study. The study addressed credibility in multiple ways. The primary researcher participated in reflexivity by using a field journal throughout the research process (Henderson & Rheault, 2004). Student success is a strong focus of the program directors job; therefore, the researcher's perception and beliefs could affect the analysis without the use of reflexivity. Credibility was also addressed through member checking and triangulation. The RAs conducted member checking following transcription and data analysis to ensure the data collected reflected the participant's accurate perceptions of their experience. Triangulation was also used among the research team to enhance the intercoder reliability of the data interpretations after the coding and recoding processes (Carter et al., 2014). To address credibility through structural coherence and STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 33 consistency among the research team members, the primary investigator developed a codebook to document emergent definitions to be used during data analysis (Henderson & Rheault, 2004). The transferability component was addressed through a thorough description of the study sample (Henderson & Rheault, 2004). The primary investigator selected the study participants based on the information they provided during recruitment. The participants were chosen to represent various demographics and attributes often found in PTA students from a program housed in a community college located in a large urban metropolitan area. The research team consisted of the primary researcher and three research assistants. To demonstrate the dependability of the study and track variability among the researchers, an audit trail of marginal notes and memoing was used during all data collection and analysis. To also address dependability, a code-recode procedure compared the results of the research team (Henderson & Rheault, 2004). Researcher bias was addressed in this study. To demonstrate the confirmability of the study, triangulation, reflexivity, and an external audit was performed. As described, triangulation was used among the research team members throughout the data analysis, and the primary investigator used a field journal to participate in a reflexive analysis (Henderson & Rheault, 2004). Results Participant (N = 11) demographics and characteristics are described in Tables 1 and 2, respectively. Participant numbers and pseudonyms were used to protect participant privacy. All participants were current students in the Houston Community College Physical Therapist Assistant (PTA) Program. Five males and six females were interviewed, with ages ranging from 23 to 43 years old. Ethnic representation of the interviewees consisted of four Caucasians, one STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 34 Hispanic, two Asian Americans, two African Americans; and two who self-reported two or more ethnicities. Six participants had been successful in all program courses while five of the participants had experienced a course failure and were required to repeat a course. Each interview was conducted through the Zoom platform and ranged from 43 to 83 minutes. Four major themes emerged that create a picture of student perception of successful performance in a PTA Program: Measures of Success, Facilitators of Success, Barriers of Success, and Program Expectations. Subthemes further elucidated the students perceptions. Theme 1: Measures of Success Measures of success emerged as an overarching theme with two sub-themes. All 11 participants referred to success in objective measures such as grades or grade point average, completing the program, passing the licensing examination, and obtaining a job as a PTA. In addition, nine of the 11 participants referred to successful performance beyond objective measures that included effective critical thinking and skill application, the ability to provide quality patient care, and intrinsic components, such as how success made them feel. One participant, Jeff, commented, obviously you need to pass but can you take what you learned and apply it to what you need to do. Another participant, John, stated, successful performance is to graduate from the program and get a job, but also to feel like you own something. Earning a grade equivalent to a C or better in all courses in the HCC PTA curriculum is required to complete the program. Students who do not earn a C or better must repeat the course and are delayed a year or are dismissed from the program. Several students commented on the importance of earning satisfactory grades but indicated a difference in what those grades needed to be. Marie related, as long as youre above the failing mark you are good while Jeff said, a good performance for me is an A that has checked all the boxes. Jennifer commented STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 35 on how her view of grades had changed over the course of the program: I used to think that getting an A was like successful performance, but I would now say a B is successful performance for me. Graduation from a CAPTE- accredited program is required for PTA students to sit for the National Physical Therapy Examination (NPTE). Passing the NPTE is required to earn a state licensure/certification which is required in all 50 states to be employed as a physical therapist assistant. John stated, get a degree, first major thing; pass the boards, second major thing; and being a licensed therapist is success. George stated, success to me is to graduate from this program and get a job. Beyond the objective measures, participants also identified successful performance in terms of critical thinking and applying skills in lab and clinical environments. Jeff stated, can we take all the information that weve just spent the last few months memorizing and then now can we move it over to apply it? Jamie commented, [success in the program] is just knowing everything like, know it and then being able to apply it like when I go to clinicals. The ability to provide quality patient care was mentioned by several interviewees as an indicator for successful performance. John stated, [successful performance is] getting people back to their functional state. George described successful performance is to watch them [patients], walk out of the clinic when they are discharged. Its heartwarming. Jordan stated, treating a patient with a T12, spinal cord injury, felt really good. That was one of my high moments in this program. The students perceptions of successful performance also included intrinsic components that provided inherent satisfaction from their successful performance such as personal positive feelings and achievements versus external grade achievements. Theo commented, to be STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 36 successful in the program to me, is to be successful in my parents eyes. Rebecca stated, being able to look back at what you have accomplished and have no regrets. Joe added, being able to have an intelligent conversation with other medical practitioners and be able to know what youre talking about. John commented [being able to be successful] in this program has made me a better person in other aspects of my life. Theme 2: Facilitators of Success Facilitators of success surfaced as a theme, which resulted in the discovery of several sub-themes that contributed to the students perception of achieving success. These sub-themes centered around classmates, faculty, mental state, family, and financial stability. All 11 participants voiced that their classmates were a large contributor to their success. Marie stated, I found people who I could study with and rely on and we kind of formed this group. Jamie added, classmates for sure have a big role in my success. Jennifer commented, classmates help because we are all stressed, and we understand what its like to be a student. Faculty were reported to be a facilitator of success by all 11 participants. Stephanie commented, I dont think I would have been able to make it this far without them telling me I am able to do it, Im smart enough, and I am not a burden. George stated, [Faculty] go way beyond the call of duty with the guys who are helping us out. Jennifer added, when things in life happen, it helps to know that the faculty supports us and dont just see us as students, but as people too. Mental state emerged as a sub-theme for facilitators of success and included categories such as mental health, work-life balance, and confidence. Nine of the 11 participants commented on the need to improve their mental health by finding balance. The actions mentioned included taking breaks or clearing their mind. Jordan stated, I spent quality time with my family. I just STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 37 decided to slow down my pace because it was overwhelming at the timescrew the books, lets go have fun. Its helped a lot with my mentality. Jeff commented, so now I know that I need to like sit back and say hey, I need time to myself, to have the best mental health. Joe added, I came to the conclusion that I do need balance, simply because I knew that it wasnt sustainable. Having self-confidence was mentioned by 10 out of the 11 participants as contributing to their mental state. Theo commented, you definitely need self-confidence to be successful. George stated, youve got to first have that mindset to say hey, I gotta be confident. I gotta be confident before I make the first step. Stephanie added, it starts with believing in yourself. Thinking you are able to achieve this goal and be successful. In relation to confidence towards a specific task, Rebecca stated being confident going into an exam or a lab practical, doing well and having your grade reflect [the performance] gives you more confidence moving forward. Family support was also voiced as a contributor to success by 10 of the participants. Jennifer commented, huge, huge, huge role [in my success]. Its my family, my boyfriend, my friends, and my church family, like they have always encouraged me. Rebecca reported, [parents] taught us to have the best mindset and make sure we do everything we do, the best we can. George added, I like family. They look up and expect success. I got a 21-year-old daughter. I gotta try to be on top of my game. The lack of financial struggles was mentioned by eight of the 11 participants to be a facilitator of successful performance. Joe mentioned, we do not have to worry about the actual tuition costs and the books cause we did preplan for all of this. Stephanie commented, Luckily I qualify for grants, so I dont have to stress. In addition, John stated, If I wasnt financially stable, I dont think I would be successful. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 38 Although there was not a specific sub-theme of external activities identified by all participants, seven of the 11 participants mentioned such activities as contributing to their success. Among these were religion (4 interviewees), exercise (3), or experience as a physical therapy technician (2). Jennifer commented, my spiritual health is very important. If Im not spending time in my faith, that does impact my self-confidence. Joe stated, I started to exercise a little more to take breaks, walk around my neighborhood. Thats been really helpful. Theo added, my mentor at work helps me be able to apply skills. You need to separate that tech mind into that PTA mindset. Being told what to do versus doing it and telling the patient what to do. Theme 3: Barriers to Success Interestingly, many of the facilitator sub-themes also surfaced as sub-themes under barriers to successful performance. Mental state with the same categories of mental health, worklife balance, and confidence rose to the top as far as barriers to success. Nine of 11 participants mentioned they struggled with balance and mental health issues that affected their success. Jordan commented, If I dont succeed at something it means that I am like nothing. Like I know nothing from the whole program. Marie described feeling overwhelmed and I think I am done with this program. Its time for me to quit. Stephanie stated, I was worried. What if I am not good enough? I dont know what my purpose is. Jeff added, There are times where it sucks because it feels like all you do is you live, eat, and breathe this stuff and there are times where you just want to go out and relax and you cant. Each of the study participants were in the PTA Program during the Coronavirus pandemic (COVID). Ten of the 11 participants mentioned that COVID and additional environmental adversities were a barrier to success. Campus closure or limited access due to weather events and COVID were mentioned. Jamie commented, Darn COVID. If COVID STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 39 didnt happen everybody would be more successful. Rebecca stated, it affects my confidence when I cannot practice at school but only at home. Stephanie added, it is draining looking at a screen all day. Im tired. You are trapped in your room all day. Another barrier to success mentioned by seven out of 11 participants was faculty behaviors and grading policies. George stated, The program doesnt seem like its student centric. Marie commented, They have a really high expectation for us, but when they [the faculty] mess up, its not the same accountability. Stephanie added, They dock you for things on your lab exams; its so overwhelming. Family dynamics, classmates, prior physical therapy technician experience, and financial burdens were mentioned by four participants as a barrier to success. Although not an overall subtheme these topics were mentioned as significant barriers by the specific participants. Stephanie stated, my dad just doesnt care, it was just so toxic with him. Jamie commented, I was really intimidated by my classmates, and I just felt they were so much better than me when it comes to labs. Theo added, Where I work as a tech, they dont do it this way [the way they were learning in lab]. This is dumb. Why is the instructor doing it this way? I didnt really want to learn that way. Rebecca stated, [lack of] finances adds more stress because I have to pay my own bills and after I totaled my car, how am I supposed to buy a new car because that costs money? Theme 4: Program Expectations Program expectations emerged as an overarching theme with three sub-themes. Participants described both externally and internally driven sources of expectations as well as how they adjusted their expectations and strategies for success after they entered the program and after exam or course failure. Ten of the 11 participants commented that the PTA program STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 40 was harder than they expected. External sources included having earned a previous degree, and the fact that the PTA program was provided in a community college, which led them to believe it was not going to be rigorous. Marie stated, I didnt think it was going to be hard. I have a degree. This was just going to be like a cool little two-year program. Stephanie commented, I didnt expect it to be so difficult. Ok, its just a community college. George expressed his surprise by stating, this program is a different ballgame. Internally driven expectations stemmed from previous personal successes and selfconfidence. John commented, I knew I was going to do well, but the program is much harder than I expected. Marie stated: Ive never failed at anything. I have a lot of self-confidence. Self-confidence is a roller coaster in this program, and it depends on the daily. You just kind of celebrate everything because you never know when it could end really in this program and thats the scary part. Adjustments to expectations and changing success strategies were mentioned by seven of the 11 participants. Out of the 11 participants, five failed a course and were required to repeat the course the following year. Rebecca stated, [failure] was a lesson I needed to learn so I could understand the work that I needed to put in to be successful. Jaime commented, I feel like I disappointed so many people when I failed, I wanted to succeed so now I spend more time studying and I cut things off with my friends and family to get rid of distractions. As rigor increased in the program, John stated, No matter how hard it is going to get, I will put in the work, and I am confident I am going to succeed. George commented, I was giving everything up, if I fail, where do I go from there, failing is not an option. Jennifer expressed, I realized STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 41 that I needed support from others to be successful [in the PTA program], after failing in the DPT program. After a disappointing exam result, Joe added, I needed to ask my classmates what theyre doing, because what I was doing was not the right way to go about it. In addition, eight of the 11 participants stated their expectations changed over time. Rebecca stated, I didnt go into it with the expectation that I would do poorly. I was very confident until I got into the program; then you realize you dont know as much as you think you did. Theo commented, I thought I was gonna do excellent. I took it for granted and underestimated the difficulty. The program got harder and harder when I expected it to get easier. George reported, I was confident. I was on top of the world until reality set in. Though the participant interviews were conducted one-on-one, the results demonstrated high consistency among the students regarding their perceptions of success, the facilitators and barriers to success that they identified and how their expectations were often different from their actual experiences in the PTA Program. See Figure 1 for a visual representation of identified themes that emerged from the study. Discussion This research study aimed to answer four questions. The primary question was to determine how students perceive successful performance in a physical therapist assistant program. Participants referred to success in objective measures such as grades or grade point average, completing the program, passing the licensing examination, and obtaining a job as a PTA. In addition, they also referred to successful performance beyond objective measures that included effective critical thinking and skill application, the ability to provide quality patient care, and intrinsic components, such as how success made them feel. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 42 The second research question was to determine how PTA students perceive the impact of external factors on their performance/success. Financial stability, positive support from classmates, faculty, and family were mentioned as external factors that contributed to successful performance in the PTA Program. Negative faculty behaviors and environmental situations that closed access to the campus such as COVID-19 and weather-related storms were mentioned as barriers to successful performance. The third research question was to determine how physical therapist assistant students perceive the influence of self-efficacy on their performance/success. As noted, Bandura (1997) defines self-efficacy as a construct within a theoretical system and confidence as a general term without specificity. During the analysis, the researchers applied the term confidence to statements made by the participants regarding their beliefs in their general ability to succeed in the program and self-efficacy when the participants referred to specific tasks such as taking a lab exam. Having self-confidence was mentioned by participants as contributing to successful performance on specific tasks and general expectations of program success. Participants also mentioned that increased self-confidence improved their overall mental state. Participants stated that when they struggled with work-life balance and mental health issues, such as stress and anxiety, their self-confidence decreased, and this change negatively affected their successful performance on exams. The fourth research question was to determine how physical therapist assistant students perception of success changed throughout the program. Participants commented that they had high expectations of success in the program initially due to their previous educational experience and past achievements. Participants expectation of success in the program changed after experiencing the rigorous educational demands found in a two-year PTA program. Although STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 43 many of the participants experienced disappointing results in examinations or final course grades, it did not adversely affect their ability to eventually succeed. Several participants commented on their need to adjust their expectations by removing distractions, seeking support, and persevering through adversity. When a student matriculates into a PTA program, it is common for them to have expectations that are based on such factors as previous work and educational experiences, their level of self-confidence and their support systems and resources. Because of the nature of the curriculum and the level of skill application required, students commonly find the need to adjust their external environment, their study strategies, and their expectations to be successful. It is the responsibility of the institution, program, and faculty to address the different needs and situations of the students in order to promote successful performance. Appropriate retention efforts should be in place to foster intellectual growth and effective coping mechanisms. Although there is no universal solution to ensure student success, an understanding of successful performance through the students perception will provide insight to inform the development of best practice strategies. The PTA students in this study identified both factors that propelled their success and barriers that inhibited their successful performance in the PTA program. Four themes emerged: measures of success, facilitators of success, barriers to success, and program expectations. All participants identified objective measures of success such as passing tests, effective lab performance, passing the National Physical Therapy Examination, and getting a job as a licensed physical therapist assistant. This is consistent with previous research that has traditionally defined student success as on-time graduation, program completion and passing the board examination (Kabiri et al., 2017; Maring et al., 2013; Wheeler et al., 2018; York et al., 2015). Similar to Bowman et al.s findings (2019), many of the participants also reported STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 44 measures of success using non-cognitive factors that included positive personal feelings and achievements such as being seen as successful in their parents eyes or receiving positive affirmation from faculty and patients. In addition, participants identified factors not found in the literature such as skill application, providing quality patient care, and being able to have intelligent conversations with other medical professionals. In light of these findings, PTA programs may benefit from retention interventions that incorporate non-cognitive factors that promote positivity, hope, and confidence such as the positive psychology intervention discussed in the research by Trask-Kerr et al. (2019). Several sub-themes were identified as facilitators of success by the participants. Classmates and faculty support and positive relationships were mentioned as facilitators by all participants. In agreement with this present studys findings, Jacobs et al. (2015) and Cassidy et al. (2020) found that positive faculty experiences, peer-to-peer emotional support, and a sense of belonging enhanced success in nursing students and United Kingdom student physiotherapists, respectively. As a retention strategy, faculty may consider the development of interventions early in the curriculum that include classroom activities that promote positivity and peer support. Another strategy that could be initiated include faculty coaching on how to deliver feedback that promotes positive emotions. Positive psychology and classroom interventions by Seligman (2009) found that positive interpersonal relationships and positive education experiences contribute to flourishing and improved quality of life. Faculty feedback can play a role in their interpersonal relationships with students. Feedback that builds on skills that are performed well increases self-efficacy and reinforces successful behaviors, while necessary corrective feedback should be centered around specific task performance. Identifying the specific performance correction promotes goal-directed behavior versus emphasizing negative emotions (Scorsolini- STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 45 Cormin et al., 2013). In addition to faculty and classmate support, family support was identified by 10 out of 11 participants as playing a role in their success. This finding is consistent with the those of Wray et al. (2014) which concluded that facilitators for nursing degree completion included a healthy family support system. PTA programs may consider improving communication and relationships with the students families by implementing a family day or a program newsletter to provide families a glimpse into the lives of PTA program students and suggest ways to support them. For the purpose of this study, mental state includes mental health, work-life balance, and confidence. Some aspect of mental state was identified by all of the participants as important to successful performance. Many felt that the need to find balance was imperative to improve their mental health, reduce stress, and enhance their focus. Participants reported that improving their mental state resulted in an increase in their overall self-confidence, which was identified as necessary to meet specific performance expectations. These findings agree with the findings authored by Van Veld et al. (2018), that self-efficacy is confidence in first year DPT students ability to use coping strategies to make a positive change in a stressful situation. A majority of our participants indicated that believing in yourself is the first step toward successful performance. This finding is consistent with Gregor (2020), who found that students in two-year public institutions are more likely to overcome barriers to achieving their goals if they believe they have the capability of succeeding. According to Burt (2021), addressing mental health issues openly by implementing mental health initiatives, which include balance, exercise and wellness, will lower stress and increase educational engagement. PTA programs can benefit by adding in work breaks during the day or classes conducted out of doors to foster a more relaxing and positive environment. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 46 Although many of the identified barriers to success involved the same categories as facilitators of success, the external environment was identified as an outlier by the participants in this study. The Coronavirus pandemic of 2020 (COVID-19) upended the educational system and has affected student opportunity and achievement. According to the Office of Civil Rights report, Education in a Pandemic: The Disparate Impacts of COVID-19 on Americas Student (2021), the pandemic has aggravated already pre-existing race- and ethnicity-based inequities in education. This finding is a concern since the majority of PTA programs reside in community colleges and Gregor et al. (2020) noted that community colleges enroll a high percentage of marginalized groups, and these students already face additional barriers to success. Ten out of 11 participants in our study indicated that the pandemic and other environmental events (such as an ice storm) that forced campus closures, negatively impacted their success. Participant comments included the inability to connect with classmates, the struggles with online learning, and the lack of opportunities to practice hands-on skill application, among others. According to the report by the Office for Civil Rights (2021), 80% of college students stated that COVID -19 negatively affected their mental health by increasing their feelings of isolation and anxiety as well as decreasing their confidence in achieving success. Burt (2021) suggests implementing strategies to combat the isolation and anxiety such as creating spaces for students to connect, transparent communication regarding the pandemic, promoting the positive aspects of remote learning, and increasing communication to students through the use of social media. Mental state was indicated by participants as a significant barrier to success, due to the challenge of maintaining work-life balance, the increased stress as a result of the rigorous coursework, and high program expectations. In related research by Gautman et al. (2020), college students reportedly faced difficulty managing the transition to college, which in turn STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 47 increased anxiety. Common comments reported by this studys participants related to self-doubt and poor health effects, such as lack of sleep. These findings are consistent with those of Ramirez et al. (2018) and Ouweneel at el. (2013) who found that high anxiety and decreased belief in university students ability can be a barrier to successful performance and goal achievement. Implementing mental health initiatives as addressed earlier, at the beginning of the PTA program, may assist students with the transition and may avoid the ill effects of increased stress and anxiety. An additional barrier to success identified by our participants was faculty behaviors of disrespect and lack of accountability. Comments included lack of student-centric behavior, mistrust of the grading policy, and a disrespectful tone when addressing students. A study by Smith et al. (2021) provides agreement with our participants comments that negative faculty attitude and behavior may impede success by reducing self-confidence and poorly affecting the student and teacher relationship. Bettencourt et al. (2021) promotes tailoring (individualizing) retention interventions with college students to build trust and that contain a holistic and studentcentered approach to enhance student success. Sharing the results of this current study with faculty may initiate conversations of how to embrace a more positive culture and improve interpersonal relationship between faculty and students. PTA program faculty may be able to promote trust in the student-teacher relationship by adding short-term goal setting activities with positive reinforcement. During these activities, faculty should use a respectful and encouraging tone to increase student self-efficacy and enhance their desire to perform successfully. The final theme centered around student expectations of success in the PTA program. Most PTA students have not experienced a technical educational program prior to matriculating in the PTA program. Expectations were derived from past experiences and inaccurate knowledge STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 48 of program rigor. Approximately 65% of Houston Community College PTA students matriculate with an earned bachelors degree; 9 of 11 of our participants entered the PTA program with a bachelors degree and one participant had an earned associate degree. Leikens (2017) research found that dental-hygiene students in a two-year program were more successful with a previous earned degree than those who lacked a previous degree. However, all participants in this study voiced their surprise at the difficulty of the PTA program which is a two-year degree offered at a community college and that the program was much harder than their previous educational experience. Comments centered around the expectation that community college would be less rigorous, especially with their previous educational experience at a four-year institution. PTA programs should consider adding real-life stories of PTA student experiences, challenges and strategies to program information sessions and program orientation. Other drivers of expectations of success were more internally situated such as perseverance, high self-confidence, and previous successes. Research on self-efficacy has determined that those individuals with high self-efficacy do not look at challenges as a threat (Bandura, 1994; Rowbotham & Schmitz, 2013). Adding a self-efficacy measurement component to program admission criteria as a non-cognitive factor may be beneficial in identifying a future at-risk student. The majority of our participants experienced a change in expectations of their success once they failed an exam or course. This resulted in the need to adjust strategies and expectations to be successful moving forward. This is consistent with literature that documents the importance of the students ability to use coping strategies to make a positive change in a stressful situation (Van Veld, 2018). Students who fail may experience low self-efficacy which can affect their determination to succeed (Ouweneel at al., 2013; Ramirez, et al., 2018). By communicating program expectations early in the process, providing activities early in the STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 49 program that allow for success, and devising interventions that promote persistence and selfefficacy following failure, students may be able to face the rigors of a professional program. In addition, these steps may foster the ability for students to bounce back following adversity. Previous literature on student success has focused on increased retention challenges in community college students due the high percentage of students who are non-traditional, come from low-economic and marginalized communities (Easley 2016; Gregor 2020). Although the participants from this present study were diverse and represented non-traditional students with financial challenges, student debt, and marginalized groups, no participant reported this as a barrier to their success. In fact, two students stated that being older gave them a better perspective on success and the effort it takes to be successful. In addition, although two participants reported over $50,000 in student debt, neither participant stated that financial issues were a barrier to their success in the PTA program. One participant commented that her lower income status while she was growing up motivated her to be successful in the PTA program so she would be able to support herself. Four of the 11 participants commented that having to work added stress to their lives and adversely affected their work-life balance but not their overall ability to be successful. Although the participants from this study represented diverse ethnicities, there was no evidence in this population of an increased risk of academic adversity based on race or ethnicity. Of the 11 participants, five experienced an educational delay due to course failure. These five participants included all races represented in the study; Caucasian (1), Hispanic (1), African American (1), and Asian (2). The six participants who experienced on-time graduation included three Caucasians, two African Americans and one student who self-reported being of two or more races. Study Limitations STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 50 This study does contain limitations. The participants were all from one PTA program. The participants represented a diversity of characteristics found in the HCC PTA program; however, most participants were self-proclaimed financially stable. In addition, the external factors experienced by the participants were varied and no one external factor seemed to have a significant effect on successful performance. Selection bias may have affected the results of the study, as students who volunteered to participate in the interviews may have had strong opinions on the subject of student success, as they persevered through course failures and were driven to share their personal experiences, both positive and negative. Although the research assistants who conducted the interviews were mentored, they did not have previous experience in qualitative interviewing. This lack of experience may have resulted in missed opportunities to develop a richer understanding of the concepts that emerged during the interviews. Although saturation was not anticipated or expected in this study, our inexperienced interviewers may not have provided adequate follow-up questions to support common activities or categories that may have approached saturation. (Creswell & Poth, 2018). A common limitation and risk in qualitative research, as noted by Henderson & Rheault, (2004), is researcher bias. The primary investigator (PI) was the director in the PTA Program under study. Although the PI did not conduct the interviews, she did perform the data analysis which may have influenced the results. To mitigate researcher bias, analysis was performed in conjunction with the research assistants, and an external audit was performed through the coding and theme development. Because the interviews were conducted virtually via the Zoom platform, this may have reduced the social interaction and visual communication between the participants and the interviewers. It may have been challenging to establish a comfortable environment and for the interviewers to see facial expressions or body language. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 51 Implications for Future Research This study serves as a foundation for future research on the student perception of successful performance. Future research could include other PTA programs, public, private, and proprietary, in differing regional locations, with different student populations, a different number of faculty, and varying cohort sizes. In addition, future research on this topic could include those students who did not succeed in the program and chose or were unable to continue. Including additional populations would allow a deeper exploration of the phenomenon under study. To develop effective retention strategies, additional research could involve investigating the impact of social interaction among classmates, as well as the effect of faculty behaviors on student success. Although Hepworth et al. (2018) found that social factors were secondary to students success in comparison to academic preparedness, the results of this current study demonstrate the need to investigate the social and intrinsic aspects of the PTA program experience. Another suggested area of research is the concept of student expectations of the PTA program experience and their potential for success. All participants in this study acknowledged that the PTA program was harder than they expected; they were not prepared for the speed of content delivery or the rigorous expectations. Additional studies could address unrealistic student expectations. Lastly, research that includes the student perception should be included in the development of both proactive and remedial retention interventions. As this research demonstrates there are many variables that contribute to student successful performance and varying lenses, including students perceptions are necessary to develop effective strategies. Conclusion STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 52 The present study summarizes four themes that pertain to perceptions of successful performance in PTA program students. The themes that emerged were measures of success, facilitators of success, barriers to success, and program expectations. The results of this study are consistent with York et al. (2015) definition of success that included academic achievement; however also included skill attainment, satisfaction, persistence, and performance following graduation. The present study results are also similar to the findings of Wray et al. (2014) who noted that facilitators of nursing student degree completion are family and peer support and clear educational expectations. Community college faculty and leaders are increasingly interested in mechanisms to become student-centered institutions, increase student retention, and improve the student experience. To enhance student success, it is imperative to meet students where they are and that cannot be done unless students are asked and heard (Ericksen, 2020). This study provides evidence that developing simple retention strategies which promote a sense of belonging and an atmosphere of positivity can be beneficial for student experience and persistence. Strategies should avoid additional burden to the student to avoid increasing mental stress and anxiety. An understanding of these student perceptions will aid program administrators and faculty in developing retention strategies that can effectively address what the students believe to be of primary importance in their ability to persist and successfully graduate in order to begin their professional lives. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 53 References Abdulghani, H. M., Al-Drees, A. A., Khalil, M. S., Ahmad, F., Ponnamperuma, G. G. & Amin, A. (2014). What factors determine academic achievement in high achieving undergraduate medical students? A qualitative study. Medical Teacher, 36, 43-48. American Physical Therapy Association (APTA). (n.d.). Physical therapist assistant education overview. Retrieved from http://www.apta.org/PTAEducation/Overview/ Bandura, A. (1994). Self-efficacy. (V. S. Ramachaudran, Ed.), Encyclopedia of human behavior 4, 71-81. Academic Press. (Reprinted in H. Friedman [Ed.], Encyclopedia of mental health. Academic Press, 1998). Bandura, A. (1997). Self-efficacy: The exercise of control. Freeman. Beer, C., & Lawson, C. (2017). The problem of student attrition in higher education: An alternative perspective. Journal of Further & Higher Education, 41(6), 773784. https://doi.org/10.1080/0309877X.2016.1177171 Bettencourt, G.M., Corwin, Z.B., Kezar, A., Kitchen, J., Estes, H., Hallett, R., & Perez, R. (2021). Tailoring programs to support student success [Brief]. USC Pullias Center for Higher Education. https://pass.pullias.usc.edu/wpcontent/uploads/2021/08/Brief_Tailoring_Programs_Aug21_final.pdf Birks, M., Chapman, Y., & Francis, K. (2008). Memoing in qualitative research: Probing data and processes. Journal of Research in Nursing, 13(1), 68-75. http://doi.org10.1177/1744987107081254 Blanco, Q. A., Carlota, M. L., Nasibog, A. J., Rodriguez, B., Saldaa, X. V., Vasquez, E. C., & STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 54 Gagani, F. (2020). Probing on the relationship between students self-confidence and self-efficacy while engaging in online learning amidst COVID-19. Journal La Edusci, 1(4), 16-25. https://doi.org/10.37899/journallaedusci.v1i4.220 Bogardus, J., Blackinton, M., Litwin, B., Morrow Nelson, T., Mitchell, K., (2021). Depression, anxiety, and stress in Doctor of Physical Therapy students: Analysis of incidence and lived experiences, Journal of Physical Therapy Education, 35(3). 251-257 https://doi.org10.1097/JTE.0000000000000185 Bowman, N. A., Miller, A., Woosley, S., Maxwell, N. P., & Kolze, M. J. (2019).Understanding the link between noncognitive attributes and college retention. Research in Higher Education, 60(2), 135152. https://doi.org/10.1007/s11162-018-9508-0 Branon, R. (2018, November 16). Learning for a lifetime. Inside Higher Ed, https://www.insidehighered.com/views/2018/11/16/why-longer-lives-require-relevantaccessible-curricula-throughout-long-careers Burt, C. (2021). 7 ways to reach at-risk college students: Research from North Carolina State and Clemson show the disparities among various groups around mental health impacts during the pandemic and how institutions can help. University Business, 24(1), 5. Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. (2014). The use of triangulation in qualitative research. Oncology Nursing Forum, 41(5), 545-547. http://doi.org10.1188/14.ONF.545-547 Cassidy, E., Norris, M., & Williams, A. (2020). What does it take to graduate? A qualitative exploration of the perceptions of successful physiotherapy graduates from one university in the UK. Physiotherapy Theory and Practice, 36(2), 316-332. https://doi.org.10.1080/09593985.2018.1485799 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 55 Cherry, L., & Blackinton, M. (2017). Student perceptions of factors influencing success in hybrid and traditional DPT programs: A Q-sort analysis. Quarterly Review of Distance Education, 18(4), 7181. Commission on Accreditation in Physical Therapy Education. (2016). PTA Standards and required elements. CAPTE accreditation portal Retrieved from http://www.capteonline.org/Faculty/AccreditedPrograms/ Commission on Accreditation in Physical Therapy Education (2020). 2019 Fact sheet: Physical therapist assistant education programs. Retrieved from http://www.capteonline.org/uploadedFiles/CAPTEorg/About_CAPTE/Resources/Aggreg ate_Program_Data/AggregateProgramData_PTAPrograms.pdf Creswell, J. W. & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches. (4th ed.). SAGE Deasy, C., Coughlan, B., Pironom, J., Jourdan, D., & McNamara, P. M., (2014). Psychological distress and lifestyle of students: Implications for health promotion. Health Promotion Internationals, 30(1), 77-87. http://dpi.org.10.1093/heapro/dau086 Denham, S. A., Tietze, M., Allam, Z., Talleff, J., Schrum, N., & Wang, T. (2018). Academic success of undergraduate nursing students. Nurse Education in Practice, 33, 172-177. https://dx.doi.org/10.1016/j.nepr.2018.08.013 Desmarais, L., Woble-Valenski, M. A., & Oestmann, E., (2011). Factors influencing physical therapist assistant licensure examination success. Journal of Physical Therapy Education, 25(2), 36-41. Dwyer, T. (2017). Persistence in higher education through studentfaculty interactions in the STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 56 classroom of a commuter institution. Innovations in Education & Teaching International, 54(4), 325334. https://doi.org/10.1080/14703297.2015.1112297 Easley, P. J. (2016). Standardized tests and GPA as predictors of academic success of physical therapist assistant students enrolled in programs at Tennessee board of regents community colleges (Publication No. 10296248) [Doctoral dissertation, The University of Memphis]. CINAHL. Ericksen, K. (2020, July 1). 8 overlooked factors affecting college persistence and retention. Collegis Education. https://collegiseducation.com/news/programs-and-coursecontent/college-persistence-and-retention/ Foo, J., Ilic, D., Rivers, G., Evans, D. J. R., Walsh, K., Haines, T. P., Paynter, S., Morgan, P., & Maloney, S. (2018). Using cost-analyses to inform health professions education - The economic cost of pre-clinical failure. Medical Teacher, 40(12), 12211230. https://doi.org/10.1080/0142159X.2017.1410123 Gautam, D., Aryal, S. C., Fernando A. D., Afza, T., Deepak, T. R., Airee, Y. S., & Joshi, D. D. (2020). Prevalence of depression, anxiety & stress among first year students in physiotherapy colleges at Bangalore, India. Indian Journal of Physiotherapy & Occupational Therapy, 14(1), 7882. https://doi.org/10.5958/0973-5674.2020.00014.3 Goldingay, S., Hitch, D., Ryan, J., Farrugia, D., Hosken, N., Lamaro, G., Nihill, C., & Macfarlane, S. (2014). "The university didn't actually tell us this is what you have to do": Social inclusion through embedding of academic skills in first year professional courses. The International Journal of the First Year in Higher Education, (5)1, 43-53. Gregor, M. A., Pino, H. V. G. D., Gonzalez, A., Soto, S., & Dunn, M. G. (2020). Understanding STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 57 the career aspirations of diverse community college students. Journal of Career Assessment, 28(2), 202218. https://doi.org/10.1177/1069072719849872 Gresham, B. B., Thompson, M., Luedtke-Hoffmann, K., & Tietze, M. (2015). Institutional and program factors predict physical therapist assistant program graduate rate and licensure examination pass rate. Journal of Physical Therapy Education, 29(4). 27-36. Hallett, R., Bettencourt, G.M., Kezar, A., Kitchen, J.A., Perez, R., & Reason, R. (2021). Reenvisioning campuses to holistically support students: The ecological validation model of student success [Brief]. USC Pullias Center for Higher Education. Henderson, R., & Rheault, W. (2004). Appraising and incorporating qualitative research in evidence-based practice. Journal of Physical Therapy Education, 18(3), 35-40. https://doi:10.1097/0000141620041000000005 Hepworth, D., Littlepage, B., & Hancock, K. (2018). Factors influencing university student academic success. Educational Research Quarterly, 42(1), 4561. Houston Community College. (n.d.). About HCC: Mission. Retrieved July 1, 2020, from https://www.hccs.edu/about-hcc/ Houston Community College (n.d.). Physical Therapist Assistant Program. Retrieved July 1, 2020, from https://www.hccs.edu/programs/areas-of-study/health-sciences/physicaltherapist-assistant/ Ingrassia J. M. (2016). Successful admission criteria to predict academic and clinical success in entry-level radiography programs. Radiologic Technology, 87(5), 502510. Jacobs, S., Atack, L., Ng, S., Haghiri-Vijeh, R., & DellElce, C. (2015). A peer mentorship program boosts student retention. Nursing 2015, 45(9). www.Nursing2015.com STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 58 Kabiri, L. S., Voight, M, Gleeson, P. B., & Mitchell, K. (2017). Predicting academic and licensure examination success among physical therapist assistant students. Journal of Physical Therapy Education, 31(4), 29-34, https://doi.org/10.1097/JTE.0000000000000014 Kebaetse, M. B., Kebaetse, M., Mokone, G. G., Nkomazana, O., Mogodi, M., Wright, J., Falama, R., & Park, E. (2018). Learning support interventions for year 1 medical students: A review of the literature. Medical Education, 52, 263-273. https://doi.org/10.1111/medu.13465 Leiken S. M. (2017). Relationship of dental hygiene students' engagement practices to their academic achievement. Journal of Dental Education, 81(10), 12131219. https://doi.org/10.21815/JDE.017.072 Linderman, D., & Kolenovic, Z. (2013). Moving the completion needle at community colleges; CUNYs accelerated study in associate programs (ASAP). Change, 45(5), 43-50. https://doi.org/10.1080/00091383.2013.824350 Malterud, K., Siersma, V. D. & Guassora, A. D. (2015). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 1-8. https://doi:10.1177/1D449732315617444 Maring, J., Costello, E., Ulfers, M., & Zuber, E. (2013). Curriculum, faculty, and cohort variables predicting physical therapist assistant program graduate success on the national physical therapy examination. Journal of Physical Therapy Education, 27(2), 33-40. Masika, R., & Jones, J. (2016). Building student belonging and engagement: Insights into higher STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 59 education students experiences of participating and learning together. Teaching in Higher Education, 21(2), 138150. https://doi.org/10.1080/13562517.2015.1122585 McComb, B. E., & Lyddon, J. W. (2016). Understanding the effectiveness and impact of student success interventions on campus. New Directions for Community Colleges, 175, 8394. https://doi.org/10.1002/cc.20214 McFadden, D. L. H. (2016). Health and academic success: A look at the challenges of firstgeneration community college students. Journal of the American Association of Nurse Practitioners, 28(4), 227232. https://doi.org/10.1002/2327-6924.12345 Mejia-Downs, A. (2020). An intervention enhances resilience in entry-level physical therapy students. Journal of Physical Therapy Education. 34(1), 2-11 https://doi.org/10.1097/JTE.0000000000000114 Merriam, S. B. (2002). Basic interpretive qualitative research. S. B. Merriam (Ed.), Qualitative research in practice: Examples for discussion and analysis (pp. 37 39). Jossey-Bass. Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publications. Mitchell, K., Ellison, J., Schaumberg, E., Gleeson, P., Bickley, C., Naiki, A., & Travis, S. (2021). Examining demographic and preadmission factors predictive of first year and overall program success in a public physical therapist education program, Journal of Physical Therapy Education. 35(3), 203-209 https://doi:10.1097/JTE.0000000000000186 National Center for Education Statistics (2020, June). Students in Sub-baccalaureate Health Sciences program: 2015-2016. U.S. Department of Education. Retrieved from https://nces.ed.gov/pubs2020/2020055.pdf STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 60 Nicholls, D. (2009). Qualitative research: Part three methods. International Journal of Therapy and Rehabilitation, 16(12), 638-647. Office for Civil Rights (2021, June 9) Education in a pandemic: The disparate impacts of COVID-19 on Americas students. Department of Education. https://www2.ed.gov/about/offices/list/ocr/docs/20210608-impacts-of-covid19.pdf Ouweneel, E., Schaufeli, W. B., & Le Blanc, P. M. (2013). Believe, and you will achieve: Changes over time in self-efficacy, engagement, and performance. Applied Psychology: Health & Well-Being, 5(2), 225247. https://doi.org/10.1111/aphw.12008 Palmadessa, A. L. (2017). Americas College Promise. Community College Review, 45(1), 52 70. https://doi.org/10.1177/0091552116673710 Ramirez, J., Isaacson, J., Smith, D., & Senger, B. (2018). Teaching life lessons: When millennials fail. Building Healthy Academic Communities Journal, 2(1), 50-59. http://doi.org/10.18061/bhac.v2i1.6360 Rowbotham, M., & Schmitz, G. S. (2013). Development and validation of a student self-efficacy scale. Journal of Nursing Care, 2(1), 1-6. http://doi.org/10.4172/2167-1168.1000126 Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin., 143(6), 565600. https://doi.org/10.1037/bul0000098 Saldana, J. (2014). The coding manual for qualitative researchers. SAGE. Satterlund, T. (n.d.). Note to self: Writing analytic memos. UC Davis Center for Evaluation and Research. http://proramevel.ucdavis.edu Seligman, M. E. P., Ernst, R. M., Gillham, J., Reivich, K., & Linkins, M. (2009). Positive STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 61 education: Positive psychology and classroom intervention. Oxford Review of Education, 35(3), 293-311. https://doi.org10.1080/03054980902934563 Scorsolini-Comin, F., Germaine Victorine Fontaine, A. M., Helena Koller, S., & dos Santos, M. A. (2013). From authentic happiness to well-being: The flourishing of positive psychology. Psicologia: Reflexo e Critica, 26(4), 663670. https://doi.org/10.1590/S0102-79722013000400006 Soika, B. (2021, September 1). What is student retention, and why does it matter? USC Rossier School of Education. https://rossier.usc.edu/what-is-student-retention-and-why-does-itmatter/ Smith, E. N., Rozek, C. S., Manke, K. J., Dweck, C. S., & Walton, G. M. (2021). Teacher versus researcherprovided affirmation effects on students task engagement and positive perceptions of teachers. Journal of Social Issues, 77(3), 751768. https://doi.org/10.1111/josi.12417 Staykova, M. P. (2012). Community college education through the looking glass of associate degree nursing. Teaching & Learning in Nursing, 7(3), 9397. https://doi.org/10.1016/j.teln.2012.01.005 Stickley, L., Bezner, J., & Spivey, S. (2019). Development and validation of the self-efficacy of student physical therapist outcomes survey. Journal of Physical Therapy Education, 33(4), 325-334. https://doi.org/10.1097/JTE.0000000000000111 Stuckey, H. L. (2015). The second step in data analysis: Coding qualitative research data. Journal of Social Health and Diabetes, 3(1), 7-10. Temi. (2020). Advanced speech recognition software. http://www.temi.com Trask-Kerr, K., Chin, T., & Vella-Brodrick, D. (2019). Positive education and the new STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 62 prosperity: Exploring young peoples conceptions of prosperity and success. Australian Journal of Education, 63(2), 190-208. http://doi.org/10.1177/0004944119860600 Turner III, D. W. (2010). Qualitative interview design: A practical guide for the novice investigator. The Qualitative Report, 15(3), 754-760. Utzman, R. R., Riddle, D. L., & Jewell, D. V. (2007). Use of demographic and quantitative admissions data to predict academic difficulty among professional physical therapist students. Physical Therapy, 87(9). 1164-1180. https://doi.org/10.2522/ptj.20060221 Van Veld, R., Slaven, E., Reynolds, B., Shupe, P, & Woolery, C. (2018). First-year doctor of physical therapy students demonstrate change in coping with stress. Journal of Physical Therapy Education, 32(2), 138-144. Verdinelli, S. & Scagnoli, N. I. (2013). Data display in qualitative research. International Journal of Qualitative Methods, 12(1), 359-381. http://doi.org10.1177/160940691311200117 Wheeler, E., Finucane, S., Soper, S., & Mayhew, T. (2018). Perceived value of preadmission physical therapy observation hours and their ability to predict academic performance. Journal of Physical Therapy Education, 32(1), 17-25 https://doi.org/10.1097/JTE.0000000000000029 Wray, J., Aspland, J., & Barrett, D. (2014). Choosing to stay: Looking at retention from a different perspective. Studies in Higher Education, 39(9), 17001714. https://doi.org/10.1080/03075079.2013.806461 York, T. T., Gibson, C, & Rankin, S. (2015). Defining and measuring academic success. Practical Assessment Research and Evaluation, 20(5), https://doi.org/10.7275/hz5x-tx03 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 63 Table 1 Participant Demographics Study Identification Number 168 180 108 152 182 175 162 114 159 122 Pseudonym Used for Study Gender Age Ethnicity Joe Jennifer Jeff Maria Stephanie Rebecca Jordan Jamie John George Male Female Male Female Female Female Female Female Male Male 38 24 24 28 24 30 35 23 37 43 Caucasian 2 or more races Caucasian Hispanic 2 or more races Caucasian Caucasian Asian Black Black STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 64 Table 2 Participant Characteristics Passed classes Pseudonym Year in YTD Married or #Children Used for PTA on 1st Household living at Study Program attempt Partner home Joe 1st year Yes Yes 1 Jennifer 1st year Yes No 0 Jeff 1st year Yes Yes 0 Maria 3rd year No Yes 2 Stephanie 3rd year No No 0 Rebecca 3rd year No No 0 Jordan 2nd Year Yes Yes 1 Jamie 3rd year No No 0 Current amount of student debt Housing Employment shares in paying my expenses not working None shares in paying 10, 000 my 10 24 expenses hours/week 29,000 shares in paying 10 24 my expenses hours/week None live with someone else who pays all my > expenses. not working 50,000 shares in paying my < 10 expenses hours/week None shares in paying my 40 or more > expenses hours/week 50,000 live with someone else who pays all my expenses. not working None live with someone 10 24 else who hours/week <10,000 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE John George Theo 2nd Year 2nd Year 3rd year Yes Yes No Yes No No 0 1 0 pays all my expenses. live with someone else who does NOT share in paying my exp shares in paying my expenses live with someone else who does NOT share in paying my exp 65 25 39 hours/week 10, 000 29,000 not working 10, 000 29,000 25 39 hours/week <10,000 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE Figure 1 66 STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 67 Appendix A Study Recruitment Information Form Hello, although you know me as Professor Voight, Program Director of the Houston Community Colleges (HCC) Physical Therapist Assistant Program (PTA), I am coming to you today as a student in the Doctor of Health Science Program at the University of Indianapolis. I have become very interested in developing strategies within the HCC PTA Program that will help students to be successful in their required courses. Although the faculty developed a retention program recently, using their wealth of experiences, we did not ask what doing well or success in the program meant to you, the PTA student. In order to fill that gap, I decided to study the PTA students perception of success and successful performance. I am here today because I want to offer you the opportunity to participate in this research study. Participation has no bearing on your status in the program or your grades. All identifiable information will remain confidential. If selected to participate, you may terminate your participation in the study at any time. This study involves the participation of three Research Assistants who are Texas Womans University (TWU) Doctor of Physical Therapy (DPT) students, and 10 PTA students from HCC. In order to participate, the PTA student must have earned a passing grade in all required program courses in at least the first semester of the program. Ten HCC PTA students will be selected from those students who volunteer. Every effort will be made to ensure a variety of participants in terms of all characteristics on the demographic form are represented in the study. Each HCC PTA student will participate in one interview conducted by two TWU DPT students (approximately 4560 minutes long). Interviews will be audio-recorded. I will not be involved in the interview process; however, I will be analyzing the transcription. You will only STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 68 be identified in the interview and the study process by your HCC student ID number. During transcription and data analysis it may be possible that I recognize a demographic characteristic of a student, but there will be no efforts made to determine the exact identity of the participant. If you are interested in participating, please complete the demographic information form that you have received. Once complete, place the form in the envelope that is being held by your lead instructor in this class. In order to participate, the demographic form must be completed by the end of class and placed in the envelope. I will not be present when you decide if you have an interest in participating. I will have no knowledge of who has or has not chosen to participate. The demographic information will be entered into a password protected spreadsheet by the program administrative assistant. She will send the spreadsheet to me and I will choose the 10 de-identified participants based on various reported demographics, as explained above. Once selected, you will be contacted by one of the TWU DPT Research Assistants through your HCC student email that you entered on the demographic form. That email will explain that you have been selected and to expect a phone call from a Research Assistant, again to the number that you indicated on your demographic form. During the phone call, the time and date for the interview will be determined. The mechanism by which the interview will be held will either be in a conference room, face to face, with two Research Assistants or it will be held via Google Meets, depending on the status of COVID-19 restrictions. The format of the interviews will be the same for all participants. The format of the interview will be decided prior to contacting you to schedule an interview. If you have any questions, I will be happy to answer them now. However, in order to maintain confidentiality and avoid any possibility of bias or coercion, questions after today STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 69 should be directed to the Research Assistants. Their names and email addresses are below. Thank you for your consideration. Sincerely, Michele Voight Research assistants: Breasha Blaylock, SPT bblaylock1@twu.edu; Kaylee Hedricks, SPT Khedricks@twu.edu Emily Mowrey, SPT emowrey@twu.edu IRB Approval Houston Community College, Texas Womans University, University of Indianapolis reliance agreement with Houston Community College STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 70 Appendix B Study Participant ID # 101 Student Demographic Information for the Qualitative Study: Student Perceptions of Successful Performance in a Physical Therapist Assistant Program 1. Participate in the study (circle one) * Yes or * No If you have circled No, you do not need to complete this demographic form. If you circled Yes, you are consenting to be a participate in the study and need to complete the following questions on this demographic form. 2. Student preferred email: _______________________________ 3. Student cell phone: _______________________________ 4. Age _____ Identified Gender _____ 5. Ethnicity (circle one) * 2 or more races * Hispanic * Caucasian * Pacific Islander * Native American * Black * Asian *Other (fill in) _____________________ 6. I am considered what year in the PTA Program (circle one) * 1st year * 2nd year * 3rd year * 4th year 7. I have completed all courses in the PTA Program to date with all passing grades on the first attempt (circle one) * yes or * no 8. Married or Household Partner (circle one) * Yes or * No 9. Number of Children living at home _____ Ages of children living at home __________ 10. Outside of day care arrangements, describe your level of assistance with caring for your children ____________________________________________________________________ ______________________________________________________________________________ STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 71 11. Housing (circle one) * I live with someone else who pays all my expenses. * I live with someone else who shares in paying my expenses. * I live with someone else who does NOT share in paying my expenses. * I live by myself and am responsible for all of my expenses. 12. Employment (circle one) * 40 or more hours/week * 25 39 hours/week * Less than 10 hours/week * not working * 10 24 hours/week * Other _________________ 13. Current amount of student debt (circle one) * More than $50,000 * $30,000 49,000 * No student debt * $10, 000 29,000 * less than $10,000 * Other ________________________ 14. Anything else that positively or negatively affects your ability to be successful in the program? All demographic information will remain confidential. The information will be used to select various characteristics that are represented in the Houston Community College Physical Therapist Assistant Program. Student names will not be used at any time. Forms will be destroyed immediately following the data entrance into an Excel demographic spreadsheet. The demographic Excel demographic spreadsheet will be destroyed following the end of the study. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 72 Appendix C Email Notification of Participant Selection Hello, Thank you for your willingness to participate in the qualitative research study, Student Perceptions of Successful Performance in a Physical Therapist Assistant Program. This email is to notify you of your selection to participate in the study. In the next two four weeks, a research assistant will call you using the phone number you provided on the demographic sheet, to schedule an interview. The mechanism by which the interview will be held will either be in a conference room, face to face, with two Research Assistants or it will be held via Google Meets, depending on the status of COVID-19 restrictions. The format of the interviews will be the same for all participants. The format of the interview will be decided prior to contacting you to schedule an interview. As a reminder, participation in the study is voluntary and has no bearing on your grades in the HCC Physical Therapist Assistant Program. We are using your school identification number for identification purposes to maintain confidentiality. You may reply to this email if you have any questions regarding the study and you may also terminate your participation in the study at any time without consequence. Thank you, Breasha Blaylock, SPT - bblaylock1@twu.edu Kaylee Hedricks, SPT - Khedricks@twu.edu Emily Mowrey, SPT - emowrey@twu.edu. Study Research Assistants STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 73 Appendix D CONSENT TO PARTICIPATE IN RESEARCH Qualitative Study: Student Perceptions of Successful Performance in a Physical Therapist Assistant Program Primary Investigator: Michele Voight, PTA, MPA voightm@uindy.edu 361-834-1159 Research Assistants: Breasha Blaylock, SPT - bblaylock1@twu.edu Kaylee Hedricks, SPT - Khedricks@twu.edu Emily Mowrey, SPT - emowrey@twu.edu. You are invited to take part in a research project to investigate the student perception of successful performance in a physical therapist assistant program. This study will be conducted by Michele Voight, PTA, MPA, Breasha Blaylock, SPT, Kaylee Hedricks, SPT, and Emily Mowrey, SPT. Purpose: The purpose of this basic interpretive qualitative study is to explore student perceptions regarding factors that influence performance in a PTA program. Procedure: This research will involve an interview where you will be asked questions by the Research Assistants (Blaylock, Hedricks, and Mowrey) related to your perceptions of successful performance in the Physical Therapist Assistant Program at Houston Community College. You will also be contacted following the interview and data analysis to clarify comments and questions that arise after analyzing the data from interview. Total Time Commitment: 45 60 minutes for the interview. Additional time of approximately 30 minutes will be required to confirm and clarify comments and interpretations following data analysis. The interview will be audiotaped and transcribed verbatim (word for word) using an encrypted transcription service. The transcription will assist the Principal Investigator (Voight) to accurately analyze the interview at a later date and to capture a more complete picture of what was said/discussed. Only the Principal Investigator and Research Assistants will have access to the transcriptions and audiotapes. Potential risks and steps to minimize the risks: 1. Loss of time. The time required will be a maximum of 60 minutes for the interview and approximately 30 minutes will be required to confirm and clarify comments and interpretations following data analysis. 2. Embarrassment, emotional or psychological distress. Interviews will be conducted in a non-judgmental manner with open-ended questions. You will have the right to take a break, to skip a question, to change the line of questioning or to terminate the interviewall without penalty. 3. Confidentiality or identity breach. Interviews will be conducted in a private meeting room (at Houston Community College (HCC) Coleman Campus in the Coleman Tower building at 1919 Pressler, Houston TX 77030). Private information will not be linked to your name. Private information will be requested only in the minimum level required to answer the research question. Raw interview transcripts and coded interviews, including demographic information and answers to interview questions and audio recordings will be accessible only to Michele Voight, Breasha Blaylock, Kaylee Hedricks, and Emily Mowrey. All data will be stored electronically STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 74 using an encrypted transcription and data analysis software application. The data will only be retrievable with the projects unique username and password. Data will be stored until the end of the study. At the end of the study, electronic files will be deleted from the software application services. 4. Coercion. Participation in this study is entirely voluntary. If you choose not to participate, there will be no penalty in terms of your physical therapist assistant (PTA) program status, your grades or any other aspect of your time in the PTA program. Confidentiality will be protected to the extent that is allowed by law. The researchers will try to prevent any problem that could happen because of this research. You should let the researchers know at once if there is a problem and they will help you. However, Texas Womans University or Houston Community College does not provide medical services or financial assistance for injuries that might happen because you are taking part in this research. 5. Participation in this study is voluntary and you may withdraw from the study at any time without penalty. 6. There are no direct benefits to you as a result of your participation in this study. 7. You will be provided with a copy of your personal interview, including codes, categories and themes used to analyze the data. Additionally, if you would like a copy of the final results of the study, inform the Principal Investigator (Voight) and it will be provided to you. You will be given a copy of this signed and dated consent form to keep. If you have any questions about the research study, you should ask the researchers; their email addresses are at the top of this form. If you have questions about your rights as a participant in this research or the way this study has been conducted, you may contact the Texas Womans University Office of Research at 713-794-2480 or via e-mail at IRB@twu.edu (Houston Community College Office of Institutional Review Board at irb@hccs.edu) Participant Name (printed) Participant signature Research Assistant signature Date Date STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 75 Appendix E Interview Questions Thank you for volunteering to be interviewed for this study. The purpose of this study is to explore student perceptions of successful performance in a physical therapist assistant (PTA) program. In this interview, we will ask you questions about your perceptions. The interview will last approximately 45 minutes to an hour. This interview is confidential; it will be audiotaped on a voice recorder. The recording will be downloaded to an encrypted transcription service. The interview will be deleted from the voice recorder as soon as the download occurs to the transcription service. The transcription will be analyzed by the primary investigator and the research assistants. The results of the transcription and analysis will be stored in a secure, password protected location, and all data will be destroyed once the project has been completed. You may ask any clarifying questions regarding the purpose of this study, interview, and how the data will be used. You may skip any question that makes you feel uncomfortable and withdraw from participation at any time without consequence. (In the case that a virtual interview is necessary due to COVID-19, and videotaping is used, the process used will be inserted with the same deletion process occurring). To ensure that we record your responses correctly, we will be audio recording the interview. Do I have your permission to do that? (once they confirm their permission) Wonderful, thank you. (start the audio recorder) I have started audio recording. Interview Questions 1. Tell us what it has been like for you being a student in the PTA program. a. Have there been any surprises? b. Is the program harder than you expected, easier, or pretty much as you expected? STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 76 2. As you know this research is about defining success in a PTA program from the students point of view. What thoughts come to mind when you think about success in this PTA program? a. Tell us how your thoughts on success in the PTA program compares to how you feel about success in other aspects in your life. b. Tell us how you perceive program faculty and classmates play a role in your definition of success in the PTA program. 3. Describe what successful performance means to you. What does that look like? a. Describe a time in the PTA program when you felt like you performed successfully. i. Talk to us how that made you feel. b. Describe a time in the PTA program when you felt like you did not perform successfully. i. Talk to us how that made you feel. c. When you had an unsuccessful performance in the PTA program, describe how you feel you were able to keep moving forward. Tell us about any strategies that you used that may have helped you bounce back. 4. Let us look at the program from a timeline. When you were notified that you were accepted into the PTA program, what went through your mind? Talk to us about that. a. Tell us how your family and friends reacted when they found out you were accepted. b. Describe how you thought you would do in the program. What thoughts went through your mind regarding your likelihood of success in the PTA program? STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 77 c. Tell us about your self-confidence heading into the PTA program. 5. So, let us move on to your PTA program orientation and that first day of the PTA program. What were your thoughts and opinions and just general feeling about that day? a. We talked about your thoughts on your potential to be successful or your selfconfidence. Did those thoughts or opinions change from the time you first heard you were accepted and following that day of orientation? 6. So, let us keep going on the timeline. Now you are in your first semester and you have your first major test. Let us come back to this idea of self-confidence. Tell us more about what you thought about your own abilities after that first test? a. To take it a step further did your perceptions of success in this PTA program change after your first test? (Explain your answer more) b. How about after the end of the first semester? c. How about after the end of the first year, the third year, or the fourth semester? d. Have you been on a clinical experience yet? Did your perceptions of success in the PTA program change after your clinical experience? e. Has there been a time elsewhere in the curriculum where you doubted your abilities? If so, describe that experience. 7. In your academic life, what do you see as the relationship between self-confidence and success? a. Provide some examples of this. 8. We want to get some ideas if the outside influences in your life and your perception on what effect those might have on either your self-confidence or your ability to be successful in general. STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 78 a. Tell us about any influence you think your family and friends may have had on your ability to be successful in the PTA program. Tell me more. b. Tell us about your perceptions of your financial situation and if that situation may have any influence on your ability to be successful in the PTA program. Tell us more. c. Talk to us about any other outside factors that you believe may influence your ability to be successful in the program. Tell us more. d. Talk to us about your school/life balance. Do you feel you have balance and describe what that means to you. (f/u explore positive and negatives). e. Thinking back to that timeline, tell us how the outside influences you discussed (financial, family, friends, work, etc) may have influenced your perception of your success or self-confidence since the beginning of the PTA program until now? 9. Let us review the topics covered, your perception of success, your self-confidence, how the other factors discussed (faculty, your classmates, family, friends, work, and social life) play a role in your success. Tell us what all has to happen or come together for you to be successful. a. Describe the most effective set-up you can imaging in your life that would help you be successful or perform successfully in the PTA program. b. Is there anything else that we didnt ask or that you didnt get to talk about that help me understand your perceptions? Thank you for your time. Your information from the interview will be de-identified and kept confidential. The audio recording of this interview will be downloaded to an encrypted STUDENT PERCEPTIONS OF SUCCESSFUL PERFORMANCE 79 transcription service. The recordings on the voice recorder or Google Meets if virtual, will be deleted once downloaded to the transcription service. The transcription will be analyzed by the primary investigator and the research assistants. The analysis will be conducted using a research application which is an encrypted service with password protection and only accessible by the primary investigator and the research assistants. Once the research project is complete, the transcripts of the interview and analysis of the data will be deleted. If you have any questions after the interview, please feel free to contact the research assistants, Breasha Blaylock, SPT at bblaylock1@twu.edu; Kaylee Hedricks, SPT at Khedricks@twu.edu; or Emily Mowrey, SPT at emowrey@twu.edu.You may also reach out to the University of Indianapolis Doctoral Committee Chair, Dr. Laura Santurri, at santurril@uindy.edu. IRB Approval - Houston Community College, Texas Womans University, University of Indianapolis reliance agreement with Houston Community College  ...

... ASSOCIATION OF FUNCTIONAL MOVEMENT AND INJURY INCIDENCE IN COLLEGIATE SOCCER ATHLETES Submitted to the Faculty of the College of Health Sciences University of Indianapolis In partial fulfillment of the requirements for the degree Doctor of Health Science By: Erin Ulrich, MS, LAT, ATC Copyright June 26, 2020 By: Erin Ulrich, MS, LAT, ATC All rights reserved Approved by: Elizabeth S. Moore, PhD Committee Chair Ed Jones, PT, DHSc Committee Member Lochana Siriwardena, PhD Committee Member __________________________________________ Accepted by: Laura Santurri, PhD, MPH, CPH Director, DHSc Program Chair, Interprofessional Health & Aging Studies University of Indianapolis Stephanie Kelly, PT, PhD Dean, College of Health Sciences University of Indianapolis FUNCTIONAL MOVEMENT AND INJURY Association of Functional Movement and Injury in Collegiate Soccer Athletes Erin Ulrich University of Indianapolis 1 FUNCTIONAL MOVEMENT AND INJURY 2 Abstract Tools such as the Functional Movement Screen (FMS) and Y-Balance Test Lower Quarter (YBT-LQ) have been used to assess quality of functional movement. To date, research has been inconclusive regarding the association between performance on the FMS and YBT-LQ and lower extremity injury incidence in collegiate soccer athletes. The purpose of this retrospective study was to explore whether functional movement, as measured by the FMS and Y-balance tests, in junction with athlete demographics, is associated with lower extremity injury. The study included retrospective data collected on 143 mens and womens soccer athletes over three years (2014-2016) from NCAA Division III Lebanon Valley College located in south-central Pennsylvania. Using chi-square tests and Mann Whitney U tests, functional movement and demographic data were compared to determine if differences existed in the data in relation to injury categories. In addition, gender differences between the FMS and YBT-LQ were explored. Gender, body mass index, injury history, FMS composite score, and YBT-LQ reach asymmetries were not associated with lower extremity injury. Additionally, the scores were not related to lower extremity noncontact injury categories. There were no significant differences between genders in YBT-LQ reach asymmetries or in the FMS CS. However, significant differences between genders were noted on ASLR and trunk stability push-up components of the FMS. Future research should seek to establish population-specific normative data for, and clinical utility of, the FMS and YBT-LQ. Clinicians should use caution in using any of these factors in isolation in clinical decision making with regard to injury prevention and return to play after injury. Keywords: functional movement, Functional Movement Screen, Y-Balance, soccer, injury, lower extremity injury, noncontact injury FUNCTIONAL MOVEMENT AND INJURY 3 Acknowledgments I would like to take this opportunity to express my gratitude and appreciation for those who have supported me along this journey. First and foremost, I could not have done this without the support of my team my children, Anna and Sam. You two have supported me in completing my degree by chipping in and displaying both empathy and responsibility. I know it has been challenging at times, but I hope it has taught you the value of hard work and dedication. Next, thank you to my parents, Randy, close friends, and colleagues. You all have been there for me in meaningful ways, and I cannot thank you enough. From helping out with the kids, to reminding me to engage in downtime, to encouraging me at work, your friendship and acts of selfless love and support have not gone unnoticed. Finally, thank you to my committee. My content expert, Dr. Jones - you brainstormed with me to develop this idea, and somehow, I convinced you to join my committee. Thank you for continuing to brainstorm with me along the way and offering your expert guidance on the topic. My analysis expert, Dr. Siriwardena - thank you for your patience and teaching. The resources you provided to me after our conversations were most helpful. And my chair, Dr. Moore - your attention to detail, research expertise, and quality of feedback was exceptional. Thank you for your mentorship and leadership. FUNCTIONAL MOVEMENT AND INJURY 4 Table of Contents Title Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Purpose Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Significance of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Chapter 2: Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Epidemiology of Soccer Injuries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Injury Risk Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Movement Screening in Sport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Movement Screening in Other Populations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Injury Prediction Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Clinical Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Chapter 3: Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 FUNCTIONAL MOVEMENT AND INJURY 5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Chapter 4: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Functional Movement Categories and Injury . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Functional Movement Categories and Noncontact Injury . . . . . . . . . . . . . . . . . . . . . 27 Athlete Factors and Injury . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Presence of Lower Extremity Injuries by Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Functional Movement and Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Chapter 5: Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Discussion of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Study Limitations and Implications for Future Research . . . . . . . . . . . . . . . . . . . . . 41 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 FUNCTIONAL MOVEMENT AND INJURY 6 List of Tables Table 1: Comparison of FMS-CS and YBT-LQ Injury Risk Categories by Injury. . . . . . . . 53 Table 2: Comparison of Frequency of FMS-CS and YBT-LQ Injury Risk Categories Between Noncontact Injury and No Injury . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Table 3: Comparison of FMS-CS scores, FMS Component Test Score Frequency, and YBT-LQ Direction Scores by Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 FUNCTIONAL MOVEMENT AND INJURY 7 List of Figures Figure 1: Data Collection Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Figure 2: Diagram of Y-Balance Test Lower Quarter Non-Stance Foot Excursions . . . . . . 59 FUNCTIONAL MOVEMENT AND INJURY 8 List of Appendices Appendix A: Lebanon Valley College Institutional Review Board Approval . . . . . . . . . . . 60 Appendix B: Lebanon Valley College and University of Indianapolis Reliance Agreement 61 FUNCTIONAL MOVEMENT AND INJURY 9 Association of Functional Movement and Injury in Collegiate Soccer Athletes It has been estimated that annually, over 210,500 injuries (6.0 per 1,000 athleteexposures) occur as a result of National Collegiate Athletic Association (NCAA) sports play. Mens and womens NCAA soccer are among the sports with the highest reported rates of injury with 8.0 and 8.4 injured per 1,000 exposures, respectively (Kerr et al., 2015 ). Of those injuries, approximately three-quarters occurred in the lower extremity (DiStefano et al., 2018; Kerr et al., 2018; Roos et al., 2017). The most frequent mechanisms for injury during mens and womens soccer practices are noncontact in nature (DiStefano et al., 2018; Kerr et al., 2018). Thus, lower extremity noncontact injuries have proven problematic in collegiate soccer. Soccer participation has grown to become among the highest in the NCAA. From 19811982 season to 2017-2018 season, the number of rostered NCAA soccer athletes increased nationally from 1,855 to 27,883 for women and from 12,957 to 25,072 for men (Irick, 2018). With participation in soccer on the rise, it is important to examine lower extremity injury risk factors in this population of athletes. The full understanding of factors associated with increased occurrence of injury remains unclear despite the frequency of noncontact injuries in mens and womens soccer athletes. Researchers have investigated the predictive ability of various factors and algorithms, including injury history, gender, and sport (Chimera, Smith, & Warren, 2015; Fousekis, Tsepis, Poulmedis, Athanasopoulos, & Vagenas, 2011). In addition, biomechanical factors such as isokinetic hamstring to quadriceps strength ratios have been studied relative to predicting lower extremity injuries (Dauty, Menu, & FouassonChailloux, 2018). Most recently, neuromuscular screening tools, including the Functional Movement Screen (FMS) and YBalance Test - Lower Quarter (YBT-LQ), aimed at assessing functional movement quality, have been investigated. Some early researchers reported promising results related to the ability of FUNCTIONAL MOVEMENT AND INJURY 10 functional movement pattern assessment to predict injury (Chorba, Chorba, Bouillon, Overmyer, & Landis, 2010; Gonell, Romero, & Soler, 2015; Huebner, Plisky, Kiesel, & SchwartzkopfPhifer, 2019; Kiesel, Plisky, & Voight, 2007; Lehr et al., 2013). Thus, it may be possible to use functional movement screens to identify deficits that may be contributing factors to increased rates of lower extremity injury, which have been troublesome in soccer. The purpose of this retrospective study was to explore whether functional movement, as measured by the FMS and Y-balance tests, in junction with athlete demographics, is associated with lower extremity injury in NCAA Division III mens and womens soccer athletes. The study addressed the following primary research question: Is there a significant difference in functional movement, as measured with the frequency of FMS composite score (FMS CS) and the YBT-LQ injury risk categories, among NCAA Division III soccer athletes who sustained a lower extremity injury versus those who did not sustain a lower extremity injury? Additionally, the following secondary questions were explored: 1. Is there a significant difference in the frequency of FMS CS and YBT-LQ injury risk categories among NCAA Division III soccer athletes who sustained a lower extremity noncontact injury versus those who did not sustain any injury? 2. Is athlete body mass indices (BMI) and history of recent injury associated with the presence and absence of lower extremity injuries in NCAA Division III soccer athletes? 3. Is there a significant difference in presence and no presence of lower extremity injuries between male and female NCAA Division III soccer athletes? 4. Is there a significant difference in functional movement, as measured by FMS CS FUNCTIONAL MOVEMENT AND INJURY 11 scores, FMS individual test score frequency, and YBT-LQ direction scores, between male and female NCAA Division III soccer athletes? A study of the association between variables such as FMS scores and YBT-LQ scores and injury is relevant to clinicians and researchers for several reasons. First, clinicians interested in reducing incidence of injury may use functional movement screening tools during the preparticipation process to screen athletes at risk and employ strategies to mitigate injury risk. Second, clinicians making return to play decisions after injury could use functional movement screens results to aid in their decision-making process as they aim to return athletes to play as quickly and safely as possible. Finally, the results from this study in functional movement screens and associations with injury add to the existing literature by examining a large cohort of mens and womens soccer athletes. Specifically, the sample size is a larger sample size than many previous studies, and the focus on one sport group eliminates preexisting confounding variables related to type of sport played. Literature Review Epidemiology of Soccer Injuries Over 210,500 annual injuries (6.0 per 1,000 athlete-exposures) are estimated to occur as a result of NCAA sport play (Kerr et al., 2015). Mens and womens soccer are among the sports with the highest number of injuries per year with 13,435 (8.0 per 1,000 athlete-exposures) and 15,113 (8.4 per 1,000 athlete-exposures) injuries respectively (Kerr et al., 2015; Roos et al., 2017). Only football and mens basketball reported a higher number of injuries annually (Kerr et al., 2015). These figures show that mens and womens soccer injuries constituted approximately 13% of the total annual injuries, as reported by 25 different sports (Kerr et al., 2015; Roos et al., 2017). As evidenced by the rates above, a higher than average injury per exposure rate was FUNCTIONAL MOVEMENT AND INJURY 12 displayed (Kerr et al., 2015; Roos et al., 2017). It is apparent that mens and womens collegiate soccer athletes are at a higher than average risk for sustaining an injury as compared to other sports. It is vital to quantify injuries and to understand the types of injuries sustained by soccer athletes. While data show specific epidemiological nuances between type and occurrence of injuries between mens and womens soccer, several general assertions can be made. First, roughly three-quarters of all reported injuries in mens and womens soccer occurred to the lower extremity during practices and competitions (DiStefano et al., 2018; Kerr et al., 2018; Roos et al., 2017). Because soccer involves a heavy emphasis on lower extremity tasks like kicking, running, cutting, and dribbling, it is not surprising that its athletes are at an increased risk of sustaining lower extremity injuries. Second, approximately half of all reported injuries were due to a combination of noncontact and overuse mechanisms of injury during practices (DiStefano et al., 2018; Kerr et al., 2018; Roos et al., 2017). Finally, athletes are frequently assessed and treated for an injury that results in no restrictions in participation. These injuries are referred to as non-time loss injuries (Dompier, Marshall, Kerr, & Hayden, 2015; Kerr et al., 2014). Often injuries that result in no restrictions on participation happen as a result of overuse or noncontact mechanisms of injury and are still treated and managed by the sports medicine staff (Roos et al., 2017). However, non-time loss injuries are underrepresented in the literature. Despite the paucity of data surrounding non-time loss injuries, they have been estimated to account for nearly half (47%) of all injuries (Roos et al., 2017). Management of non-time loss injuries incur increases in time, attention, and resources from a medical standpoint. Thus, it may be vital to include injuries that do not result in time loss when considering factors associated with noncontact mechanisms FUNCTIONAL MOVEMENT AND INJURY 13 of injury. Injury Risk Factors Within the sports of mens and womens soccer, lower extremity noncontact muscle injuries are common (DiStefano et al., 2018; Kerr et al., 2018; Roos et al., 2017). However, it may be possible to identify risk factors for lower extremity injuries, which are troublesome to soccer athletes. While injury-causing events involving contact with another player or object may not be modifiable, factors contributing to noncontact injury events may prove modifiable. In particular, identifying modifiable risk factors and strategies aimed at reducing incidences of injury is of interest to many stakeholders such as athletes, coaches, athletic trainers, strength and conditioning coaches, and researchers. Intrinsic and extrinsic risk factors may play a role in the occurrence of noncontact injuries. Extrinsic risk factors include type and quality of the playing surface, footwear, and environmental factors such as rain and field grade (Orchard, Seward, McGivern, & Hood, 2001). Intrinsic factors, or player characteristics, that have been shown to contribute to injury risk include variables such as height and weight (Fousekis et al., 2011), aerobic fitness (Lisman, OConnor, Deuster, & Knapik, 2013), injury history (Fousekis et al., 2011; Orchard et al., 2001), and muscle strength and flexibility factors (Dauty et al., 2018; Fousekis et al., 2011). Many types of intrinsic variables are collected routinely by medical staff in NCAA athletics and may have associations with injury risk. However, incorporation of movement screening as part of the preparticipation examination is not routine (Cook, Burton, & Hoogenboom, 2006). Consideration of the association between various regularly collected variables in junction with movement screening could provide valuable insight into injury risk factors. Movement Screening in Sport FUNCTIONAL MOVEMENT AND INJURY Athlete safety and readiness to participate in sport have long been assessed through medical pre-participation examinations and subsequent sports performance testing. However, assessment of fundamental movement patterns required for sport-specific movements is lacking. Tests such as the FMS and YBT-LQ are assessment tools that may fill the gap between medical clearance and sport performance testing (Cook et al., 2006; Plisky et al., 2009). The FMS consists of a battery of seven individual movement pattern tests, each with a possible score ranging from 0-3 for a total possible score of 21 (Cook, 2015). It was developed as a means of standardizing and measuring functional movement patterns and combines factors such as stability, functional asymmetries, and dynamic balance (Cook, 2015; Cook et al., 2006). The practical intent of the FMS is for clinicians to have a means of screening to identify movement dysfunction as well as establishing a standardized means of communication between sports medicine and performance enhancement teams (Cook, 2015; Cook et al., 2006). Similarly, the YBT-LQ, derived from the Star Excursion Balance Test, assesses for functional movement and dynamic balance deficits to the lower extremity (Plisky et al., 2009). While the emphasis on creating screening tools aimed at identifying movement dysfunction may serve as a guide for clinicians in their practice, it is unclear if quality of functional movement leads to increased injury incidence. Researchers interested in identifying injury risk factors quickly gained interest in studying the association between functional movement screens such as the FMS and YBT-LQ and injury due to some early studies. Kiesel et al. (2007) were the first to publish promising results on the topic; they reported an eleven-fold increased risk of injury in a cohort of professional football athletes with an FMSCS of 14 or less out of 21 possible. This landmark study established the cutoff commonly used by clinicians of 14 and was supported by Chorba et 14 FUNCTIONAL MOVEMENT AND INJURY al. (2010), who found a fourfold increase in injury risk using the same cutoff for a group of NCAA Division II female soccer, volleyball, and basketball athletes. Garrison et al. (2015) further supported the cut score of 14, reporting a 15-fold increase in injury risk in a group of NCAA Division I varsity and club athletes from a variety of unspecified sports. Similarly, some researchers reported that greater than a 4 cm bilateral reach variance in the YBT-LQ resulted in an increased likelihood of injury (Gonell et al., 2015; Smith, Chimera, & Warren, 2015). However, research shows inconsistent findings when it comes to an association between increased injury incidence and FMS and YBT-LQ composite scores. Some researchers found a greater association with individual battery test scores or right versus left-sided asymmetries rather than composite scores on the FMS and YBT-LQ (Fousekis et al., 2011; Mokha, Sprague, & Gatens, 2016; Warren, Smith, & Chimera, 2015). For example, Mokha et al. (2016) found no increased likelihood of injury associated with FMS composite score. However, they did identify an increased likelihood associated with any recorded asymmetry or individual test scores of one or less (out of a possible three). Meanwhile, other researchers found increases in predictive ability of the FMS composite scores only when combined with other risk factors such as injury history (Garrison et al., 2015) or age (Cosio-Lima et al., 2016). Still, several researchers found no association between injury and the FMS (Dorrel, Long, Shaffer, & Myer, 2018; Lisman, Hildebrand, Nadelen, & Leppert, 2019; Smith & Hanlon, 2017; Walbright, Walbright, Ojha, & Davenport, 2017; Warren et al., 2015) or YBT-LQ (Lai et al., 2017; Lisman et al., 2019; Walbright et al., 2017; Wright, Dischiavi, Smoliga, Taylor, & Hegedus, 2017) in collegiate athletes. The association between incidences of injury and variables such as performance on functional movement screens remains unclear in the literature. Additional studies are warranted to identify trends that either support or refute associations between injury incidence and 15 FUNCTIONAL MOVEMENT AND INJURY 16 functional movement. Movement Screening in Other Populations While the FMS and YBT-LQ are primarily geared toward traditional athletic groups, other physically active populations have taken an interest in these tests. The military is a large, physically active population for which the prospect of injury reduction via identification of modifiable risk factors has garnered attention (de la Motte, Gribbin, Lisman, Beutler, & Deuster, 2017; Lisman et al., 2013; Teyhen, 2014). Similar to baseline screening done during the PPE process in athletics, the military has used various functional movement screens, including the FMS and YBT-LQ, as part of the entrance examination process or physical fitness testing (de la Motte et al., 2017; Lisman et al., 2013). Specific aims of these initiatives include establishing population specific norms (de la Motte et al., 2017; Teyhen, 2014), reliability testing (Shaffer et al., 2013), and injury prediction (Cosio-Lima et al., 2016; Lisman et al., 2013). Lisman et al. (2013) found increases in injury occurrence when FMS CS was combined with other variables such as aerobic fitness. Meanwhile, Cosio-Lima (2016) reported associations between increased injury incidence with both increased age and lower FMS CS (< 14) but no association with YBTLQ scores. Occupational medicine also has a vested interest in exploring functional movement screening, particularly in fields requiring a high level of physical activity such as first responders and performing arts professionals. Butler et al. (2013) identified an increased association between injury incidence and both the FMS CS as well as two of the seven individual FMS components (deep squat and trunk stability push-up) in a group of firefighters during training. In performing arts, recent studies have been aimed at seeking associations between various functional movement screening tests and loading forces, which are considered precursors to FUNCTIONAL MOVEMENT AND INJURY 17 many overuse dance injuries (Armstrong, 2020; Armstrong, Brogden, Milner, Norris, & Greig, 2018). Injury Prediction Challenges Identification of methods aimed at predicting injuries is challenging to establish due to the complexity of confounding variables. Basic demographics such as age (Cosio-Lima et al., 2016; Warren et al., 2015), previous history of injury (Chorba et al., 2010; Fousekis et al., 2011; Garrison et al., 2015; Warren et al., 2015), gender and sport (DiStefano et al., 2018; Dompier et al., 2015; Eckard, Kerr, Padua, Djoko, & Dompier, 2017; Kerr et al., 2015; Kerr et al., 2018; Roos et al., 2017; Tenan, 2016), and anthropometric data such as height, weight, and body mass index (BMI) (Fousekis et al., 2011; Warren et al., 2015) have been reported to play a role in increased injury incidence. Of note, the association between increased injury incidence and FMS score reported by Chorba et al. (2010), was only supported when patients with a history of anterior cruciate ligament repair (ACLR) were excluded from the data. This exclusionary finding suggests a possible positive association between history of injuries and the role of subsequent rehabilitation with higher FMS scores. While history of injury is thought to increase injury incidence, in this case, the authors suggested that the significant lower extremity strength, conditioning, and neuromuscular training involved in post ACLR rehabilitation, may have resulted in increased FMS scores and decreased injury incidence (Chorba et al., 2010). Since improved performance on the FMS has been demonstrated via an injury prevention intervention program (Huebner et al., 2019), it is possible that post-injury rehabilitation can improve a patients functional movement to above baseline and, in turn, decrease injury risk. An additional consideration includes the causality of interacting variables. For example, a history of injury and specific FMS components were found to be related. However, it is not clear if movement FUNCTIONAL MOVEMENT AND INJURY 18 dysfunction(s) caused the initial injury or if the effects of the injury resulted in subsequent movement dysfunction (Chimera et al., 2015). Limitations identified in previous studies include consideration of confounding variables. For instance, all published studies that compared gender differences included a variety of sports, many of which do not have comparable opposite gender counterparts such as football (mens) and field hockey (womens) (Chimera et al., 2015; Lai et al., 2017; Mokha et al., 2016). No known studies have sampled a male and female population who exclusively play the same sport as a means of controlling for sport differences between genders, such as soccer. Soccer, commonly played worldwide, maintains consistent rules and style of play between mens and womens soccer, allowing for greater confidence in findings relative to gender differences. The prevalence of soccer around the world and consistency of play between genders makes it an ideal sport to research. Another limitation noted involves the operational definition of injury used in each study. Some researchers included only significant injuries, as defined by varying amounts of playing time lost (Dauty et al., 2018; Fousekis et al., 2011; Gonell et al., 2015; Kiesel et al., 2007; Lai et al., 2017; Lisman et al., 2019; Walbright et al., 2017). For the NCAAs Injury Surveillance System, an epidemiological database gathering data from many NCAA institutions, a reported injury was operationalized to include only time-loss injuries. However, this definition was also noted to be a limitation in capturing a significant number of injuries that result in no time loss but still require medical attention and resources (Dompier et al., 2015; Kerr et al., 2014; Roos et al., 2017). Thus, other researchers defined injury to include minor injuries or non-time loss injuries (Chorba et al., 2010; Dorrel et al., 2018; Lisman et al., 2013; Mokha et al., 2016; Smith & Hanlon, 2017; Warren et al., 2015). FUNCTIONAL MOVEMENT AND INJURY 19 Additional limitations in previous studies include small sample sizes (Chorba et al., 2010; Dossa, Cashman, Howitt, West, & Murray, 2014; Lisman et al., 2019; Mokha et al., 2016; Walbright et al., 2017; Warren et al., 2015), sport inconsistency between gender groups (Chimera et al., 2015; Lai et al., 2017; Mokha et al., 2016; Warren et al., 2015), a short length of injury surveillance (Smith & Hanlon, 2017), and inability to account for confounding variables (Chimera et al., 2015; Garrison et al., 2015; Kiesel et al., 2007; Lai et al., 2017; Mokha et al., 2016; Walbright et al., 2017). The numerous limitations identified in research studies to date add to the rationale that more studies are needed in this area. Injury prediction variables have proven too numerous and diverse to tackle with consistent results in a few landmark studies. As a result, researchers parse-out related independent and dependent variables into smaller, more manageable studies. Although the current literature is extensive and rich, it does not paint a complete picture. Researching the sport of soccer and factors associated with common injuries that occur within the sport, improves the generalizability of the study and may have clinical applications. Clinical Application Functional assessment screening tools aim to bridge the gap between PPEs and sportspecific performance testing (Cook et al., 2006). Tests designed to fill this void supply clinicians with a screening tool that has two main practical applications. First, practitioners involved in injury prevention, have a particular interest in identifying modifiable injury risk factors. In the case of poor functional movement, clinicians seek to implement interventions aimed at correcting dysfunctional movement in order to decrease injury risk (Chorba et al., 2010; Gonell et al., 2015; Huebner et al., 2019). Second, clinicians involved in rehabilitation and return to play decisions after an injury occurs can use functional movement screens as a guide to symmetrical FUNCTIONAL MOVEMENT AND INJURY 20 and optimal movement quality and as a benchmark in the process of returning to sport participation (Chorba et al., 2010; Doherty et al., 2015; Gonell et al., 2015; Kiesel et al., 2007; Lehr et al., 2013). In order to make informed decisions, clinicians need to understand the clinical utility of such tools as well as have access to population-specific normative data (Agresta, Slobodinsky, & Tucker, 2014; de la Motte et al., 2017; Teyhen, 2014) More evidence is needed to enable clinicians to make decisions on how best to utilize the FMS and YBT-LQ tools in their practice. In addition, access to normative data representative of specific populations is critical to decisions surrounding injury prevention and return to sport after injury. Therefore, this retrospective cohort study aims to fill some critical gaps in the literature, contributing to the larger picture of the role of functional movement screens in identification of injury risk. Method Study Design This was a non-experimental study using a retrospective cohort study design. The study took place at Lebanon Valley College (LVC) in the Athletic Training department, where the data are stored. The study occurred between January 2020 and April 2020. Prior to data collection, institutional review board approval was obtained from LVC, (Appendix A) and a reliance agreement was established between LVC and University of Indianapolis (Appendix B). Participants A convenience sample of 143 athletes, including all rostered NCAA Division III mens and womens soccer team members at LVC, was included in this study. Inclusion criteria included all rostered athletes who remained on the active roster for the 2014-2015, 2015-2016, and 2016-2017 seasons. At the time of testing, athletes were asked if they had sustained a FUNCTIONAL MOVEMENT AND INJURY 21 significant musculoskeletal injury or concussion in the previous six months that caused or (in the case of offseason) would have caused loss in playing time. Exclusion criteria included athletes who were not medically cleared for unrestricted athletic participation at the time of testing. Data Collection Data from the academic years 2014-2015, 2015-2016, and 2016-2017 were extracted from Athletic Trainer System (ATS), the electronic health record used to record athlete data, from patient paper charts, and from the colleges network drive. Lower extremity injury (yes/no) and mechanism (contact/noncontact) was collected from ATS. For each athlete the following data were collected from either the athletes paper chart or from the colleges network drive depending on the storage method used for athlete health records: the maximum of each of three YBT-LQ directions (cm), lower limb length (cm), individual and composite FMS scores, gender (male/female), age, height, weight, and recent injury history. See Figure 1 for organizational flow of data collection sources. Operationalization of variables. For the purpose of this study, injury was defined as any musculoskeletal injury that occurred as a result of participation in an NCAA team-related practice or competition and required medical attention or the athlete sought advice from a certified athletic trainer or other health care provider. Lower extremity injuries included all injuries to the hip and distal. Noncontact injuries were those injuries that occurred in the absence of contact with a player, ball, or equipment, as recorded by the evaluating athletic trainer. Functional movement was measured by the FMS and YBT-LQ tests. High-risk and low-risk categories were defined using the common cut scores for each functional movement test. For FMS, high-risk was defined as those individuals who scored a 14 or less (Kiesel et al., 2007). For YBT-LQ, high-risk was defined as those individuals who scored a 4.0 cm or higher normalized FUNCTIONAL MOVEMENT AND INJURY 22 reach distance asymmetry in any direction (Gonell et al., 2015). Instruments Functional movement screen. The FMS is a screening tool used to evaluate functional movement in a physically active population. It consists of a battery of seven individual tests, each with a possible score ranging from 0-3 for a total possible score of 21 (Cook et al., 2006). A score of zero is assigned if pain was present at any time during that test or with any positive clearing tests. A score of one is given if the participant was unable to complete the movement pattern as instructed. A two is assigned if the movement was completed via compensatory movement strategies. And a score of three indicates correct completion of the movement pattern as instructed (Cook et al., 2006). The most commonly accepted cutoff score for the FMS CS is 14, where those with an FMS CS of 14 or less are considered to be at a higher injury risk (Kiesel et al., 2007). Good interrater (ICC = .81 [95% CI, .70, .92]) and intrarater (ICC = .81 [95% CI, .69, .92]) reliability have been pooled and reported in military and athlete populations for the FMS (Bonazza, Dhawan, Smuin, Onks, & Silvis, 2017). Y-balance lower quarter. The YBT-LQ is a tool used to assess single-leg dynamic balance. Derived from the star excursion balance tests, it assesses reach distance in cm of the non-stance foot in three directions: anterior, posteromedial, and posterolateral (See Figure 2 for diagram of directional excursions of non-stance foot). Normalized reach difference is calculated and recorded by dividing the reach distance by the limb length (Gonell et al., 2015; Lai et al., 2017). A validated cutoff score for right versus left normalized reach distance difference on the YBT-LQ is 4.0 cm, where those with a YBT-LQ of 4.0 cm or more left to right reach asymmetry are considered to be at a higher injury risk (Gonell et al., 2015). Good interrater test-retest FUNCTIONAL MOVEMENT AND INJURY reliability (ICC (2, 3) = .85-.93), measurement error (SEM = 2.0 - 3.5 cm and MDC values = 5.5 - 9.7 cm) have been demonstrated in an active adult (military) population for the YBT-LQ (Shaffer et al., 2013). Procedures Data collection. Each year the LVC athletic trainers collected baseline functional movement screen data via the YBT-LQ and FMS during the annual pre-participation examinations in August. The head athletic trainer and a collaborating physical therapy faculty member held certifications in movement screenings using the FMS. Together, the two members of the sports medicine team at LVC, held annual trainings for the assessors prior to data collection to learn, review, and practice the test protocols on live subjects. The test assessors consisted of clinical athletic trainers and physical therapy students. Before testing, each athlete completed a questionnaire consisting of demographics, current and recent injury history, and current medical restrictions. The YBT-LQ protocol involves a limb length measurement from the inferior portion of the Anterior Superior Iliac Spine to the most inferior portion of the medial malleolus allowing for body relative calculations of right and left limb composite scores, or normalized scores. The YBT-LQ protocol requires each subject to maintain limb stance while performing three excursions with the non-weight bearing limb in the anterior, posteromedial, and posterolateral directions (Plisky et al., 2009). Reach distances in each direction were recorded for all participants. Individual FMS tests were organized and administered in seven stations, one for each test: the deep squat, hurdle step, inline lunge, shoulder mobility, active straight-leg raise, trunk stability push-up, and rotary stability. As per FMS protocol, three clearing tests (for shoulder 23 FUNCTIONAL MOVEMENT AND INJURY 24 mobility, trunk stability, and rotary stability) were scored as + or -, indicating if pain was present. If the clearing test accompanying the individual FMS test was scored as a + (pain present), the individual test was assigned a score of zero. Each tester administered and scored one movement screen. The athletic trainers recorded all injuries that occurred during the subsequent season in ATS, the electronic health record used to record athlete data. Injuries were recorded by mechanism (contact/noncontact), location on the body (i.e., shoulder, knee, thigh), and type (i.e., sprain, strain, fracture). Data extraction. Data extraction began by seeking athletes who met the inclusion and exclusion criteria. The official mens and womens soccer rosters for each year included in the study were obtained from the LVC athletics department. The rosters included only those athletes who remained on the team for the entirety of the regular season. Athletes who sustained a season-ending injury during the season but remained on the team were not removed from the official rosters and were included in the study. Any player not medically cleared for soccer at the time of testing was not included in the testing and therefore was not included in the study, even if they returned to play during the season. The participant codes were cross-referenced with their paper and electronic records, using the codebook as a guide. The independent variable data were extracted and recorded in Excel. The dependent variable or injury data were extracted from ATS via ATSs data mining feature, which automatically populates selected injury data onto an Excel spreadsheet. The data was directly recorded onto one master Excel document. Data management. Athlete names were initially deidentified using a numeric, random code generator. The names and codes were maintained in a password protected Excel document. FUNCTIONAL MOVEMENT AND INJURY 25 All athlete data were then entered into a separate master Excel document using the athlete numeric codes. While data were being extracted, electronic records were recalled on the primary investigators encrypted and password-protected computer. All original paper charts were accessed and kept in their original locations. All information was entered into Excel directly from the original paper charts from their storage locations. The password-protected Excel file containing deidentified data mined from ATS was stored in the researcher's encrypted local drive to maintain an audit trail. Data Analysis Data analysis was completed using IBM SPSS Statistics for Windows, Version 24.0 (IBM Corp., Armonk, NY). Normality of data was determined with the Shapiro-Wilk test. All tests were two-tailed, and a family-wise significance level of less than .05 was considered statistically significant. Descriptive statistics were conducted for the entire sample. Nominal data are reported as frequencies and percentages. Ordinal data and non-normally distributed interval and ratio data are reported with medians and interquartile ranges. To determine if there were significant differences in functional movement scores by gender and by BMI, Mann-Whitney U tests were conducted on data that were not normally distributed (YBT-LQ) and for ordinal data (FMS CS scores). To determine if the pre-established high-risk and low-risk categories for the FMS and YBT-LQ were associated with whether or not a player sustained a lower extremity injury and a noncontact lower extremity injury, chisquare tests were conducted. Fisher exact tests are reported when any counts were less than five. Post-hoc tests were conducted using a Bonferroni correction as needed to determine pairwise differences. Odds ratios (OR) are reported for all two-sided chi-square and Fisher exact tests. To determine if athlete BMI or gender were associated with lower extremity injuries, Mann- FUNCTIONAL MOVEMENT AND INJURY 26 Whitney U (for BMI) and chi-square tests (for gender) were conducted. Recent injury history (yes/no) data were only available for 2015. Data from this subgroup of 39 athletes were explored using Fisher exact tests and odds ratios are reported to determine its role as a confounding variable. Three participants had isolated data points that were either missing or reflected suspected recording errors. These data were handled as missing data via the default removal of them by SPSS. Finally, two athletes scored a zero on the shoulder mobility test. To maintain three groups in cross tabulation, these two scores were collapsed into one group. Results Data were collected on 143 NCAA Division III soccer athletes. The percentage of male and female athletes was similar (49.7%, 50.3%, respectively). Basic descriptive statistics of study athletes included the following means (SD); age (years) 19.36 (1.23), height (inches) 67.36 (3.81), weight (pounds) 150.90 (22.99), BMI 23.27 (2.34). Of the 89 total lower extremity injuries, the most common types were muscle/tendon injuries n = 59 (34.3%), contusions n = 45 (26.2%), and sprains n = 18 (10.5%). Functional Movement Categories and Injury The primary research question asks, is there a significant difference in functional movement, as measured by the frequency of FMS CS and YBT-LQ injury risk categories among NCAA Division III soccer athletes who sustained a lower extremity injury versus those who did not sustain a lower extremity injury? Descriptive statistics and comparison results can be found in Table 1. When considering FMS CS and YBT-LQ risk categories, no differences existed between the injured and uninjured groups. Specifically, results indicated that the frequency of FMS CS injury risk categories did not differ statistically between those who were injured and those who were not injured at an alpha level of .008. For the YBT-LQ injury risk categories: FUNCTIONAL MOVEMENT AND INJURY 27 anterior risk, posterolateral risk, and posteromedial risk, no differences existed between the injured and uninjured groups at a family-wise alpha of .05. Confidence level of the statistical results was adjusted using Bonferroni correction to account for the 6 hypothesis tests. (p = 1.00, p = .036, p = .728, respectively). Overall, there was not a significant difference in functional movement among NCAA Division III soccer athletes who sustained a lower extremity injury versus those who did not sustain an injury. Functional Movement Categories and Noncontact Injury The first secondary research question asks, is there a significant difference in FMS CS categories among NCAA Division III soccer athletes who sustained a noncontact lower extremity injury versus those who did not sustain an injury? Functional movement and noncontact injury data can be found in Table 2. When considering FMS and YBT-LQ risk categories, no differences existed between the noncontact injury and not injured groups. Specifically, for the FMS CS, results indicated that the groups did not differ statistically at an alpha level of .008 (p = .711). For the YBT-LQ anterior high-risk group (p = .557), the YBT-LQ posterolateral high-risk group (p = .036), and the YBT-LQ posteromedial high-risk group (p = .703), results indicated that the groups did not differ significantly at an alpha of .008. There was not a significant difference in functional movement categories among NCAA Division III soccer athletes who sustained a lower extremity noncontact injury versus those who did not sustain an injury. Athlete Factors and Injury The second secondary research question asks, are athlete the factors body mass index (BMI) and history of recent injury associated with lower extremity injuries in NCAA Division III soccer athletes? FUNCTIONAL MOVEMENT AND INJURY 28 Body mass index. BMI between the injured and uninjured groups for all lower extremity injuries were compared. The median (interquartile range [IQR]) BMI was 22.93 kg/m2 (3.26) for the uninjured group and 22.71 kg/m2 (2.43) for the injured group indicating the groups did not differ statistically (Z = -1.29, p = .196) at an adjusted alpha of .008. BMI between the injured and uninjured groups for noncontact lower extremity injuries was compared. The median BMI was 22.71 kg/m2 (2.35) for the noncontact injuries group and 22.93 kg/m2 (3.26) for the uninjured group. Results indicated that the groups did not differ statistically (Z = -1.35, p = .179) at an adjusted alpha of .008. History of recent injury. An exploratory subgroup of 39 athletes was analyzed to determine whether recent injury history was associated with injury category. The analysis revealed that 5 of 8 (62.5%) athletes who reported previous recent injury were subsequently injured compared to 27 out of 39 (69.2%) who reported no recent injury history. The two-sided Fishers exact test indicated no statistically significant difference (p = .697) in recent injury history and athletes who were injured or not injured at an alpha level .05. The OR was calculated to = 0.74; 95% CI [.15, 3.61]. This indicated that those who had a recent injury history were 1.4 times more likely to be injured than those who did not have a recent history of injury. Presence of Lower Extremity Injuries by Gender The third secondary research question asks, is there a significant difference in presence or no presence of lower extremity injuries between male and female NCAA Division III soccer athletes? For all lower extremity injuries, 44 of the 71 (62.0%) males were injured compared to 45 out of 73 (61.6%) females. The two-sided Fishers exact test indicated no statistically significant difference (p = 1.00) between genders at a family-wise alpha level .05. The odds ratio (OR) was calculated to = 0.99; 95% CI [.50, 1.93]. This indicates that females were 1.01 times FUNCTIONAL MOVEMENT AND INJURY 29 more likely to have a lower extremity injury than males in NCAA Division III soccer. For noncontact lower extremity injuries, males were equally likely to have a noncontact injury, 30 of the 57 (52.6%), compared to 31 out of 59 (52.5%) females. The two-sided Fishers exact test indicated no statistically significant difference (p = 1.00) between genders at a familywise alpha level .05. The OR = 0.99; 95% CI [.48, 2.07]. This indicates that males were < 1.01 times more likely to have a noncontact lower extremity injury than females in NCAA Division III soccer. Functional Movement and Gender The final secondary research question asks, are there significant differences in functional movement as measured by the FMS CS, frequency of FMS individual test scores and the YBTLQ reach difference scores for each direction between male and female NCAA Division III soccer athletes? The YBT-LQ anterior reach difference score, the YBT-LQ posterolateral reach difference score, YBT-LQ posteromedial reach difference score, results indicated that scores did not differ statistically by gender at a family-wise alpha of .05 (p = .699, p = .782, p = .489, respectively). Confidence level was adjusted using Bonferroni correction to account for the 4 hypothesis tests. For the FMS CS, it was determined that a significant difference did not exist between genders on FMS CS (p = .023) at a family-wise alpha of .05. There was not a significant difference in functional movement, as measured by FMS and Y-balance scores between male and female NCAA Division III soccer athletes. An exploratory analysis of gender difference on each FMS component was completed. Two components, the ASLR and the trunk stability push-up, displayed differences between genders. For the ASLR, it was determined that females are significantly more likely to be in the higher category than males (p = .001) at an alpha of .05. Post-hoc pairwise group differences FUNCTIONAL MOVEMENT AND INJURY 30 revealed that females were 19.8 times more likely than males to achieve a score of 3 versus a score of 1 (p < .001). Gender differences between scores of 1 and 2 (p = .023), and 2 and 3 (p = .033) were not significant at an adjusted alpha of .017. For the trunk stability push-up, it was determined that males were more likely to be in a higher category than females (p < .001) at an alpha of .05. Post-hoc pairwise group differences revealed that males were 10.9 times more likely than females to achieve a score of 2 versus a score of 1 (p < .001), males were 3.5 times more likely than females to achieve a score of 2 versus a score of 3 (p = .005), and females were 3.1 times more likely than males to achieve a score of 1 versus a score of 3 (p = .013). There was a significant difference in functional movement, as measured by the ASLR and trunk stability components of the FMS, between male and female NCAA Division III soccer athletes. For the remainder of the FMS components, the deep squat (p = .776), hurdle step (p = .145), inline lunge (p = .098), shoulder mobility (p = .095), and rotary stability (p = .224), the data indicated that the groups did not differ statistically at an alpha of .05. There was not a significant difference in functional movement, as measured by the deep squat, hurdle step, inline lunge, shoulder mobility, and rotary stability components of the FMS, between male and female NCAA Division III soccer athletes. Data pertaining to functional movement and gender is displayed in Table 3. Discussion In this study, multiple research questions were explored to determine whether functional movement, as measured by the FMS and YBT-LQ, in junction with athlete demographics, is associated with lower extremity injury in NCAA Division III mens and womens soccer athletes. The questions sought to determine whether or not significant differences existed in: (1) functional movement, as measured by the FMS and YBT-LQ, between male and female soccer FUNCTIONAL MOVEMENT AND INJURY 31 athletes, (2) the rate of lower extremity injuries between male and female soccer athletes, (3) athlete factors and injury categories, (4) functional movement test scores and injury categories, and (5) functional movement among soccer athletes who sustained a lower extremity noncontact injury versus those who did not sustain an injury. Functional Movement Categories and Injury With regard to functional movement and injury, this study sought to determine whether or not significant differences existed in functional movement test scores and lower extremity injuries. Functional movement, as measured by the FMS and YBT-LQ, was not associated with lower extremity injury in NCAA Division III mens and womens soccer athletes in this study. The FMS (Bonazza et al., 2017; Shultz, Anderson, Matheson, Marcello, & Besier, 2013) and YBT-LQ (Garrison et al., 2015; Plisky et al., 2009) are reliable tools. Further, it has been suggested that performance on functional movement tools such as the FMS and YBT-LQ are associated with increased injury incidence. Early studies showed promising findings with relation to the predictive ability of these tests (Butler, Contraras et al., 2013; Butler, Lehr, Fink, Kiesel, & Plisky, 2013; Chorba et al., 2010; Garrison et al., 2015). However, more recently published studies may suggest otherwise (Dorrel et al., 2018; Lai et al., 2017; Lisman et al., 2019; Mokha et al., 2016; Walbright et al., 2017; Wright et al., 2017). One limitation in early studies involves heterogony of sport. Since various sports have nuanced repetitive movements and subsequently varied injury epidemiology, it is plausible that including a variety of sports may not adequately capture sport-specific relationships that may exist between functional movement and injury. This study, with a specific soccer population, allowed for increased control of this type of extraneous variable in an attempt to better isolate sport-specific differences. The limitation in doing this is that it evaluated a homogeneous group who were FUNCTIONAL MOVEMENT AND INJURY 32 engaging in similar training regimens. Thus, a group was created with little variation. A debated factor in studies involving injury is the operationalizing of injury. Traditionally researches defined injury based on various levels of time-loss, reserving only injuries resulting in lost playing time for study inclusion (Butler, Contreras, et al., 2013; Garrison et al., 2015; Lai et al., 2017; Lisman et al., 2019; Mokha et al., 2016; Walbright et al., 2017). However, some authors have broadened the definition of injury to exclude time-loss as a factor in their definition of injury (Chorba et al., 2010; Lisman et al., 2013; Warren et al., 2015; Wright et al., 2017). Dompier et al. (2015) found non-time-loss injuries to account for over 80% of all reported injuries in high school sports. Accordingly, it can be inferred that, for an athletic trainer, non-time-loss injuries represent a majority of the workload. Similarly, it is possible that athletes continuing to play with an injury are not playing at full capacity. This study operationalized definition of injury was broad to represent a larger constituency of injured athletes. Nevertheless, the inclusion of minor injuries may have diluted any relationship between functional movement and more severe injuries resulting in time-loss. Three interesting trends can be seen in previous studies concerning the role injury inclusion criteria play in the relationship between FMS and injury. First, studies that resulted in a positive relationship between FMS and injury existed whether the researchers defined injury through a broad or narrow definition considering time-loss. In four studies employing a broad injury definition (including all medically evaluated injuries regardless of time-loss), one reported a relationship between FMS score and injury (Chorba et al., 2010), and one reported no relationship (Warren et al., 2015). The third reported only a relationship where FMS, when added to their three-mile run time, strengthened the relationship found between run time and injuries (Lisman et al., 2013). The second trend was demonstrated in two studies that included FUNCTIONAL MOVEMENT AND INJURY 33 three distinctive injury definitions in their samples and found similar relationships regardless of their definition (Dorrel et al., 2018; Lisman et al., 2013). Specifically, regardless of injury definition, FMS alone was not predictive of injury. The third trend was noted in a meta-analysis where it was determined that when all studies were included, the relative risk (RR) of those at high-risk was 1.51, 95% CI [1.35, 1.69]. Comparatively, for those studies that included timeloss as part of their injury definition, a slightly lower RR was calculated for those in the highrisk category, RR=1.44, 95%CI [1.05, 1.99] (dos Santos Bunn, Rodrigues, & da Silva, 2019). These differences would indicate that when less significant injuries are included in the study, the RR is higher, but it is unclear if the severity of injury included in the data affects the relationship between FMS score and injury. For clinicians seeking guidance in making return to play decisions, standardized tools aimed at reducing the risk of reinjury are helpful and desired. Many field-expedient physical performance tests have strong construct validity regarding utility in discriminating between injured and uninjured limbs, aiding clinicians in making return to play decisions post-injury (Hegedus, McDonough, Bleakley, Baxter, & Cook, 2015). It is common practice amongst clinicians to seek bilateral symmetry as a measure of return to baseline for an injured extremity. Clinicians often compare segment or limb function bilaterally, when possible, to determine what is normal based on the contralateral segment or limb. However, this study found that a greater than 4 cm reach difference left to right on the YBT-LQ was not associated with increased injury occurrence. Though early studies identified and established the commonly used 4 cm cut-points as significant with relation to injuries, the lack of significant associations found in this study, are in agreement with more recent studies (Lai et al., 2017; Lisman et al., 2019; Wright et al., 2017). Therefore, it can be surmised that reach asymmetry alone may not be associated with increased FUNCTIONAL MOVEMENT AND INJURY 34 injury incidence. Further, Fousekis et al. (2011) found no increased incidence of hamstring injury in a similar group of soccer athletes with left to right asymmetries in flexibility, proprioception, midthigh girth, and knee laxity. Only asymmetrical functional leg length was positively associated with increased incidence of hamstring injuries. In many sports, such as soccer, left to right asymmetries are not only inherent but fostered simply by a dominant sided preference in the athlete or by nature of the sport, position, and sport movements. For example, in soccer, a majority of athletes are right foot dominant. As a result, they will often prefer to plant on the left foot while kicking with the right. The plant leg is subsequently likely to develop differences that affect the entire kinetic chain. The findings of this study support the idea that, while asymmetries are not protective, they may not be harmful to an athlete. The demands of the sport and common areas of injury for that sport should be considered when making injury prevention programs and return to play decisions rather than purely relying on achievement of bilateral symmetry. Concerning clinical relevance, it is important to note that in both sport and occupational preventative medicine, clinicians should evaluate the common strains and demands of a particular sport, position, or occupation. Each one is likely to require an in-depth understanding of and unique approaches to injury prevention. Respectively, trends related to associations between various movement screens or other risk factors and injury should be sought. It is vital to the clinicians working with specific groups to have access to research or knowledge specific to their population. Functional Movement Categories and Noncontact Injury Finally, with regard to functional movement and noncontact injuries, this study sought to determine whether or not significant differences existed in functional movement among soccer FUNCTIONAL MOVEMENT AND INJURY 35 athletes who sustained a lower extremity noncontact injury versus those who did not sustain an injury. In this study, functional movement, as measured by the FMS and YBT-LQ, was not associated with lower extremity noncontact injury in NCAA Division III mens and womens soccer athletes. Similar to the debate about inclusion of non-time-loss or less severe injuries in prior studies of injury incidence, is the decision to exclude contact injuries as they are less likely to be avoidable. Therefore, they are less likely to be modifiable via preventative intervention measures. Since contact injuries may be less preventable, noncontact injuries may have a more significant association with various modifiable risk factors. On the contrary, it can be argued that contact injuries would be less likely to occur, or less severe, if an individual has quicker reaction time to avoid contact, and improved strength and dynamic stability to withstand the contact. The literature remains largely unclear about the association between functional movement and noncontact lower extremity injuries. While this study showed no significant difference between either test and injury, it is essential to view the literature concerning the relationship between FMS and YBT-LQ with injury individually. In prior studies, the FMS CS was found to have no association with incidence of lower extremity noncontact injury (Smith & Hanlon, 2017; Warren et al., 2015) or overuse injury (Lisman et al., 2013). Only one study found a significant difference between FMS injury risk categories and injury incidence (Lehr et al., 2013). The YBT-LQ is potentially more promising than FMS when it comes to identification of an association between YBT-LQ scores and incidence of lower extremity noncontact injuries. Butler et al. (2013) reported an initial relationship in this area using a composite score cut-point. Similar findings were reported by Gonell et al. (2015), who specifically looked at both contact and noncontact injuries. Interestingly, in their sample of professional and amateur soccer athletes, the following FUNCTIONAL MOVEMENT AND INJURY 36 components of the YBT-LQ were significantly different for the noncontact injuries groups but were not significantly different in the contact injuries groups: posteromedial reach difference, lower than average posteromedial reach distance, lower than average posterolateral reach distance, and lower than average composite reach scores. These findings, exclusive to the noncontact group, indicate that noncontact injuries may have a stronger relationship to specific components of the YBT-LQ than contact injuries. In this study, no association was found between functional movement, as measured by FMS CS and YBT-LQ risk categories and noncontact injuries. This finding suggests that functional movement is not associated with contact nor noncontact mechanisms of injury in NCAA Division III soccer athletes. Athlete Factors and Injury The next research questions involved athlete characteristics and injury rate. Specifically, this study sought to determine whether or not significant differences existed between BMI and lower extremity injuries. In this study, BMI was not associated with lower extremity injury in NCAA Division III mens and womens soccer athletes. Lower BMI was previously identified as a risk factor for injuries in a group of Division I athletes (Warren et al., 2015). Since this study sampled multiple sports, it is possible that this finding was dependent on sport. For example, in a collision sport, such as football, a smaller player may be at increased risk of injury when colliding with a larger player. Of the 195 sampled in the study by Warren et al. (2015), the largest number of participants played football (n = 74). Wright et al. (2011) also surveyed a variety of sports, but did not include football and found no association between BMI or sport and injury. Therefore, it may be unlikely to see sport-specific relationships between BMI and injury when certain sports are included, and aggregate data is reported. FUNCTIONAL MOVEMENT AND INJURY 37 Given that BMI is composed of height and weight, it is possible that they could have a canceling effect when factored together. For example, Fousekis et al. (2011) found increased weight and decreased height to be risk factors for quadricep strains in a group of professional soccer athletes. However, others have reported no association between height, weight, or BMI and injury (Dauty et al., 2018; Wright et al., 2017). One additional variable that was explored in this study due to its role as a possible confounding variable was previous injury history. In a subset sample (2016 data), self-reported injury history of the six months prior to testing was collected. The significance, type, or location of injury was unknown to the researchers. In this subsample group, previous injury history had no significant association with subsequent injury. It has been reported that previous injury history was associated with future injury (Chorba et al., 2010; Fousekis et al., 2011; Garrison et al., 2015; Warren et al., 2015). While injury is traditionally thought to increase the risk of future injuries (Garrison et al., 2015; Warren et al., 2015), Chorba et al. (2010) identified a protective effect between post ACLR patients and injury incidence. This protective effect was supported by Fousekis et al. (2011), who reported an increased incidence of hamstring injuries in soccer athletes who had no history of a hamstring injury. It is thought that this protective effect could be a result of rehabilitation undergone by the athlete after injury (Chorba et al., 2010). Unlike the studies mentioned above, we did not collect a detailed or extended injury history. Instead, only a recent patient-reported injury history was collected. These results revealed that recent injury history was likely not a confounding variable for the sample at large. Injury Category by Gender Concerning gender, this study sought to determine whether or not significant differences existed in the rate of lower extremity injuries between male and female soccer athletes. In this FUNCTIONAL MOVEMENT AND INJURY 38 study, injury rate did not vary between gender groups. Prior studies have also shown a similar injury rate between male and female athletes (Roos et al., 2017; Wright et al., 2017). Similar to this study, both of the aforementioned studies included a broad definition of injury, which comprises all injuries evaluated by medical staff regardless of type, severity, or time-loss. Injury rate differences are noted in two studies looking at specific types of injuries. Tenan (2016) reported a higher rate of male emergency department (ED) visits as a result of sport and recreation. It is reasonable to conclude that injuries warranting an ED visit were more significant and were unlikely to resume normal activity the following day. In the second study, only noncontact injuries were included, and females were reported to have a higher injury rate than males (Warren et al., 2015). Thus, the definition of injury and inclusion criteria could directly affect injury rates for genders. This study included a broad definition of injury; however, it was specific to location of the injury as it included all lower extremity injuries that occurred during practice or competition and for which medical attention was sought. Severity or time-loss was not used to differentiate significance of injuries. Non-time-loss injuries have been reported to account for approximately 81% of female and 83% of male high school soccer injuries (Dompier et al., 2015). A significant proportion of injuries that are evaluated in the athletic training clinic are non-time-loss injuries. Therefore, it may be important to include a broader injury definition to obtain a more comprehensive picture of the types and rates of injuries experienced across genders in future studies. In addition to severity or time-loss, rates between genders could vary based on sport. In the sport of soccer, researchers have found similar overall injury rates between genders, regardless of injury definition (DiStefano et al., 2018; Kerr et al., 2015; Kerr et al., 2018; Roos et al., 2017). FUNCTIONAL MOVEMENT AND INJURY 39 Functional Movement and Gender This study sought to determine whether or not significant differences existed in functional movement, as measured by the FMS and YBT-LQ, between male and female soccer athletes. Composite scores on the FMS between males and females were not found to be significantly different in female versus male soccer athletes in this study. This finding is consistent with many prior studies showing no difference between genders (Agresta et al., 2014; Chimera et al., 2015; de la Motte et al., 2017). Only Teyhen et al. (2014) reported higher mean FMS-CS in females younger than 30 years of age as compared to males in the same age group. However, a paucity of data exists specific to the sport of soccer regarding differences between functional movement and gender. An exploratory analysis of components of these functional movement screening tools was conducted to give a more comprehensive picture of gender differences for this specific population. While few previous studies have reported differences in the FMS-CS between genders, some have reported differences in components of the screen among physically active populations. The two significant differences between genders for components of the FMS supported in this study were for the ASLR and trunk stability push-up. First, these data revealed that the ASLR, a measure of range of motion, or flexibility about the hip, were significantly different between genders. Namely, more males scored in the lowest category of 1 on the ASLR compared to females while more females scored in the highest category of 3 compared to males. Agresta el al. (2014) reported a significantly higher Active Straight-Leg Raise (ASLR) score in a group of recreational female distance runners compared to their male counterparts. This higher female ASLR score was supported by Chimera et al. (2015) in a group of NCAA Division I athletes. In addition, Chimera et al. (2015) reported a higher shoulder mobility score and inline FUNCTIONAL MOVEMENT AND INJURY 40 lunge score in females. However, in the Chimera et al. (2015) study, it is possible that sport differences accounted for some of the variability reported between genders as some of the athletes sampled did not have an opposite gender equivalent sport to control for that variable. The findings of significantly different female scores for tests involving mobility are consistent with researchers suggestion that females generally exhibit more flexibility than males (Baechle & Earle, 2008). Researchers of previous studies also reported significant differences between genders on the trunk stability push-up. Trunk stability push-up scores were reported as higher in males compared to females (Agresta et al., 2014; Chimera et al., 2015). In this study, significant differences were noted between genders. The post-hoc analysis revealed an interesting trend in the sample. While more females had a score of 1 than males and more males had a score of 2 than females, slightly more females had a score of 3 than males. This trend indicates that males had more symmetry in this test, while females had more disparity. While the trunk stability pushup is purported to be exclusively a measure of core stability, compromised upper body strength is listed as an implication (Cook, 2015). Another test of core stability, the rotary stability test, showed similar scores between genders in this study, indicating that male and female soccer athletes have similar core stability. Thus, the disparity in trunk stability push-up scores between genders may be related to upper body strength differences achieved by some females via strength training. While this finding of no difference in the rotary stability component was in agreement with one study (Agresta et al., 2014), it was inconsistent in another (Chimera et al., 2015). Due to the nature of the trunk stability push-up, upper body strength may play a more significant role in the test than previously thought. Females have a lower absolute strength compared to males (Bishop, Cureton, & Collins, 1987). Since the trunk stability push-up aims to measure core FUNCTIONAL MOVEMENT AND INJURY 41 stability, the protocol accounts for these upper body strength differences between genders via a modification for females in an attempt to mitigate the role that upper body strength plays in this component. This modification may have been enough for some trained females to overcome the scores of their male counterparts but not enough for a majority of women. These findings indicate that modification for females made in the protocol for this component may not be enough to account for gender differences in upper body strength. It appears that gender differences in upper body strength may be a confounding variable for this components utility as a measure of core stability. This study showed no significant differences between genders when considering individual reach distances in the YBT-LQ three directions of left to right reach asymmetries. In each of the reach directions (anterior, posterolateral, and posteromedial), asymmetries were generally not found to be different between genders in prior studies. Only the anterior reach direction was found to be significantly lower in females in one study (Chimera et al., 2015). For the FMS, two of the test components are purported to assess dynamic stabilization: the hurdle step and inline lunge. In both of these components, male and female mean scores were not significantly different in this study. This finding of no significant relationship between genders on the YBT-LQ, along with the similar mean scores on the inline lunge and hurdle step, would suggest that dynamic balance is similar between male and female soccer athletes. Limitations There are some limitations of this study. While homogeneity of the sample group can add to the literature from the standpoint of sport-specific norms and relationships, it makes the results less generalizable to the physically active population at large. Previous discussion was made surrounding the operationalization of injury and its impact on study results. It is also important to FUNCTIONAL MOVEMENT AND INJURY 42 highlight this studys inclusion of a broad injury definition, which has the potential to dilute the results if we had included only time-loss injuries. The data is unclear if a narrower inclusion criterion strengthens the relationship. As a retrospective study, we did not have access to severity or time-loss criteria in the data to include multiple definitions of injury. However, we were able to look at both noncontact and all injuries. In addition, adoption of a broad injury definition is not unusual. Retrospective study designs come with inherent limitations due to various threats (Tofthagen, 2012). For example, in using retrospective data, we were bound by the data collected. In addition to severity and time-loss data not being recorded, limited information about injury history and the primary data collection methods is available to the researchers. In addition, retrospective data may not be free of bias. It is also unknown if historical or maturation threats were present. Huebner et al. (2019) demonstrated that injury risk category could be improved via an eight-week intervention program. Similarly, Sprague et al. (2014) identified changes in FMS scores throughout the competitive season. In this study, it is not known if any intentional or unintentional interventions were present between August test date and injuries that occurred toward the end of the season in November. If an unknown intervention was present or an intervention in the form of a regular competitive season, such an intervention could have resulted in a change in functional movement and skewed the data collected closer to the end of the season. This study only evaluated a relationship between functional movement and injury via previously established cut scores. A few researchers have established FMS CS cut-points other than 14 (dos Santos Bunn et al., 2019). In addition, researchers have selected various measures to evaluate relationships between YBT-LQ and injury (Butler, Lehr, et al., 2013; Gonell et al., FUNCTIONAL MOVEMENT AND INJURY 43 2015; Gribble, Hertel, & Plisky, 2012). Thus, even though we selected common cut-points, it is possible these were not optimal cut-points for these data. Conclusion Results of this study indicate that FMS and YBT-LQ scores are not related to injury in NCAA Division III soccer athletes. Additionally, the scores were not related to whether or not a noncontact injury occurred. Caution should be used in employing the FMS and YBT-LQ as injury screening tools or tools to determine return to play readiness after an injury. Further, gender, BMI, and recent injury history were not associated with whether or not an injury would occur in this sample. While no significant differences in YBT-LQ between males and females were found, some differences were noted in FMS scores between genders. No significant difference was found between genders on the FMS CS. However, significant differences between genders were noted on two components of the FMS: ASLR and trunk stability push-up. More research is needed to establish sport-specific norms for and clinical utility of the FMS and YBTLQ. FUNCTIONAL MOVEMENT AND INJURY 44 References Agresta, C., Slobodinsky, M., & Tucker, C. (2014). Functional Movement Screen Normative values in healthy distance runners. International Journal of Sports Medicine, 35(14), 1203-1207. doi:doi:10.1055/s-0034-1382055 Armstrong, R. (2020). The relationship between the functional movement screen, star excursion balance test and the Beighton score in dancers. The Physician and Sportsmedicine, 48(1), 1-10. doi:10.1080/00913847.2019.1624658 Armstrong, R., Brogden, C., Milner, D., Norris, D., & Greig, M. (2018). Functional movement screening as a predictor of mechanical loading and performance in dancers. Journal of Dance Medicine & Science, 22(4), 203-208. doi:10.12678/1089-313X.22.4.203 Baechle, T. R., & Earle, R. W. (2008). Essentials of strength and conditioning, (3rd ed.). Champaign, IL: Human Kinetics. Bishop, P., Cureton, K., & Collins, M. (1987). Sex difference in muscular strength in equallytrained men and women. Ergonomics, 30(4), 675-687. doi:https://doi.org/10.1080/00140138708969760 Bonazza, N. A., Dhawan, A., Smuin, D., Onks, C. A., & Silvis, M. L. (2017). Reliability, validity, and injury predictive value of the Functional Movement Screen. American Journal of Sports Medicine, 45(3), 725-732. doi:10.1177/0363546516641937 Butler, R., Contreras, M., Burton, L., Plisky, P., Goode, A., & Kiesel, K. (2013). Modifiable risk factors predict injuries in firefighters during training academies. Work (reading, Mass.), 46(1), 11-17. doi:10.3233/WOR-121545 Butler, R. J., Lehr, M. E., Fink, M. L., Kiesel, K. B., & Plisky, P. J. (2013). Dynamic balance performance and noncontact lower extremity injury in college football players: An initial FUNCTIONAL MOVEMENT AND INJURY 45 study. Sports Health, 5(5), 417-422. doi:10.1177/1941738113498703 Chimera, N. J., Smith, C. A., & Warren, M. (2015). Injury history, sex, and performance on the functional movement screen and y balance test. Journal of Athletic Training, 50(5), 475485. doi:10.4085/1062-6050-49.6.02 Chorba, R. S., Chorba, D. J., Bouillon, L. E., Overmyer, C. A., & Landis, J. A. (2010). Use of a functional movement screening tool to determine injury risk in female collegiate athletes. North American Journal of Sports Physical Therapy, 5(2), 47-54. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2953387/ Cook, G. (2015). Functional Movement Screen: Level 1. Retrieved from https://www.functionalmovement.com/files/Articles/717a_650a_FMS%20Level%201%2 0Online%20V1%203-21-2016.pdf Cook, G., Burton, L., & Hoogenboom, B. (2006). Pre-participation screening: The use of fundamental movements as an assessment of function Part 1. North American Journal Sports Physical Therapy, 1(2), 62-72. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2953313/ Cosio-Lima, L., Knapik, J. J., Shumway, R., Reynolds, K., Lee, Y., Greska, E., & Hampton, M. (2016). Associations between functional movement screening, the y balance test, and injuries in Coast Guard training. Military Medicine, 181(7), 643-648. doi:10.7205/MILMED-D-15-00208 Dauty, M., Menu, P., & FouassonChailloux, A. (2018). Cutoffs of isokinetic strength ratio and hamstring strain prediction in professional soccer players. Scandinavian Journal of Medicine & Science in Sports, 28(1), 276-281. doi:10.1111/sms.12890 de la Motte, S. J., Gribbin, T. C., Lisman, P., Beutler, A. I., & Deuster, P. (2017). The FUNCTIONAL MOVEMENT AND INJURY 46 interrelationship of common clinical movement screens: Establishing population-specific norms in a large cohort of military applicants. Journal of Athletic Training, 51(11), 897904. doi:10.4085/1062-6050-51.9.11 DiStefano, L. J., Dann, C. L., Chang, C. J., Putukian, M., Pierpoint, L. A., Currie, D. W., . . . Kerr, Z. Y. (2018). The first decade of web-based sports injury surveillance: Descriptive epidemiology of injuries in US high school girls' soccer (2005-2006 through 2013- 2014) and National Collegiate Athletic Association women's soccer (2004-2005 through 20132014). Journal of Athletic Training, 53(9), 880-892. doi:10.4085/1062-6050-143-17 Doherty, C., Bleakley, C. M., Hertel, J., Caulfield, B., Ryan, J., & Delahunt, E. (2015). Laboratory measures of postural control during the star excursion balance test after acute first-time lateral ankle sprain. Journal of Athletic Training, 50, 651-664. doi:http://dx.doi.org/10.4085/1062-6050-50.1.09 Dompier, T. P., Marshall, S. W., Kerr, Z. Y., & Hayden, R. (2015). The National Athletic Treatment, Injury and Outcomes Network (NATION): Methods of the surveillance program, 20112012 through 20132014. Journal of Athletic Training, 50(8), 862-869. doi:10.4085/1062-6050-50.5.04 Dorrel, B., Long, T., Shaffer, S., & Myer, G. D. (2018). The Functional Movement Screen as a predictor of injury in National Collegiate Athletic Association Division II athletes. Journal of Athletic Training, 53(1), 29-34. https://doi.org/10.4085/1062-6050-528-15 dos Santos Bunn, P., Rodrigues, A. I., & da Silva, E. B. (2019). The association between the functional movement screen outcome and the incidence of musculoskeletal injuries: A systematic review with meta-analysis. Physical Therapy in Sport, 35, 146-160. doi:10.1016/j.ptsp.2018.11.011 FUNCTIONAL MOVEMENT AND INJURY 47 Dossa, K., Cashman, G., Howitt, S., West, B., & Murray, N. (2014). Can injury in major junior hockey players be predicted by a pre-season functional movement screen A prospective cohort study. The Journal of the Canadian Chiropractic Association, 58(4), 421-427. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4262798/ Eckard, T. G., Kerr, Z. Y., Padua, D. A., Djoko, A., & Dompier, T. P. (2017). Epidemiology of quadriceps strains in National Collegiate Athletic Association athletes, 20092010 through 20142015. Journal of Athletic Training, 52(5), 474-481. doi:10.4085/10626050-52.2.17 Fousekis, K., Tsepis, E., Poulmedis, P., Athanasopoulos, S., & Vagenas, G. (2011). Intrinsic risk factors of noncontact quadriceps and hamstring strains in soccer: A prospective study of 100 professional players. British Journal of Sports Medicine, 45(8), 709-714. Garrison, M., Westrick, R., Johnson, M., & Benenson, J. (2015). Association between the functional movement screen and injury development in college athletes. International Journal of Sports Physical Therapy, 10(1), 21-28. Retrieved from https://uindy.on.worldcat.org/oclc/5737748520 Gonell, A. C., Romero, J. A., & Soler, L. M. (2015). Relationship between the y balance test scores and soft tissue injury incidence in a soccer team. International Journal of Sports Physical Therapy, 10(7), 955-966. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/26673848 Gribble, P. A., Hertel, J., & Plisky, P. (2012). Using the Star Excursion Balance Test to assess dynamic postural-control deficits and outcomes in lower extremity injury: A literature and systematic review. Journal of Athletic Training, 47(3), 339-357. doi:10.4085/10626050-47.3.08 FUNCTIONAL MOVEMENT AND INJURY 48 Hegedus, E. J., McDonough, S. M., Bleakley, C., Baxter, D., & Cook, C. E. (2015). Clinicianfriendly lower extremity physical performance tests in athletes: A systematic review of measurement properties and correlation with injury. Part 2the tests for the hip, thigh, foot and ankle including the star excursion balance test. British Journal of Sports Medicine, 49(10). doi:10.1136/bjsports-2014-094341 Huebner, B. J., Plisky, P. J., Kiesel, K. B., & Schwartzkopf-Phifer, K. (2019). Can injury risk category be changed in athletes? An analysis of an injury prevention system. International Journal of Sports Physical Therapy, 14(1), 127-134. doi:10.26603/ijspt20190127 Irick, E. (2018). Student-athlete participation 1981-82 - 2018-19: NCAA sports sponsorship and participation rates report. Retrieved from https://ncaaorg.s3.amazonaws.com/research/sportpart/201819RES_SportsSponsorshipParticipationRatesReport.pdf Kerr, Z. Y., Dompier, T. P., Snook, E. M., Marshall, S. W., Klossner, D., Hainline, B., & J., C. (2014). National Collegiate Athletic Association Injury Surveillance System: Review of methods for 20042005 through 20132014 data collection. Journal of Athletic Training, 49(4), 552-560. doi:10.4085/1062-6050-49.3.58 Kerr, Z. Y., Marshall, S. W., Dompier, T. P., Corlette, J., Klossner, D. A., & Gilchrist, J. (2015). College sports-related injuries - United States, 2009-10 through 2013-14 academic years. MMWR: Morbidity & Mortality Weekly Report, 64(48), 1330-1336. doi:10.15585/mmwr.mm6448a2 Kerr, Z. Y., Putukian, M., Chang, C. J., DiStefano, L. J., Currie, D. W., Pierpoint, L. A., . . . Marshall, S. W. (2018). The first decade of web-based sports injury surveillance: FUNCTIONAL MOVEMENT AND INJURY 49 Descriptive epidemiology of injuries in US high school boys' soccer (2005-2006 through 2013- 2014) and National Collegiate Athletic Association men's soccer (2004-2005 through 2013-2014). Journal of Athletic Training, 53(9), 893-905. doi:10.4085/10626050-166-17 Kiesel, K., Plisky, P. J., & Voight, M. L. (2007). Can serious injury in professional football be predicted by a preseason functional movement screen? North American Journal Sports Physical Therapy, 2(3), 147-158. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2953296/ Lai, W. C., Wang, D., Chen, J. B., Vail, J., Rugg, C. M., & Hame, S. L. (2017). Lower quarter ybalance test scores and lower extremity injury in NCAA Division I athletes. Orthopaedic Journal of Sports Medicine, 5(8), 1-5. doi:10.1177/2325967117723666 Lehr, M. E., Plisky, P. J., Butler, R. J., Fink, M. L., Kiesel, K. B., & Underwood, F. B. (2013). Field-expedient screening and injury risk algorithm categories as predictors of noncontact lower extremity injury. Scandinavian Journal of Medicine & Science in Sports, 23(4), e225-e232. Retrieved from http://proxylvc.klnpa.org/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=s3h&A N=89023931&site=ehost-live Lisman, P., Hildebrand, E., Nadelen, M., & Leppert, K. (2019). Association of functional movement screen and y-balance test scores with injury in high school athletes. Journal of Strength and Conditioning Research. Advance online publication. doi:10.1519/JSC.0000000000003082 Lisman, P., OConnor, F. G., Deuster, P. A., & Knapik, J. J. (2013). Functional movement screen and aerobic fitness predict injuries in military training. Medicine & Science in Sports & FUNCTIONAL MOVEMENT AND INJURY 50 Exercise, 45(4), 636-643. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5675370/ Mokha, M., Sprague, P. A., & Gatens, D. R. (2016). Predicting musculoskeletal injury in National Collegiate Athletic Association Division II athletes from asymmetries and individual-test versus composite functional movement screen scores. Journal of Athletic Training, 51(4), 276-282. doi:10.4085/1062-6050-51.2.07 Orchard, J., Seward, H., McGivern, J., & Hood, S. (2001). Intrinsic and extrinsic risk factors for anterior cruciate ligament injury in Australian footballers. The American Journal of Sports Medicine, 29(2), 196-200. Retrieved from https://uindy.on.worldcat.org/oclc/118788976 Plisky, P. J., Gorman, P. P., Butler, R. J., Kiesel, K. B., Underwood, F. B., & Elkins, B. (2009). The reliability of an instrumented device for measuring components of the star excursion balance test. North American Journal Sports Physical Therapy, 4(2), 92-99. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2953327/ Roos, K., Wasserman, E., Dalton, S., Gray, A., Djoko, A., Dompier, T., & Kerr, Z. (2017). Epidemiology of 3825 injuries sustained in six seasons of national collegiate athletic association men's and women's soccer (2009/2010-2014/2015). British Journal of Sports Medicine, 51(13), 1029-1034. doi:10.1136/bjsports-2015-095718 Shaffer, S. W., Teyhen, D. S., Lorenson, C. L., Warren, R. L., Koreerat, C. M., Straseske, C. A., & Childs, J. D. (2013). Y-balance test: A reliability study involving multiple raters. Military Medicine, 178(11), 1264-1270. doi:10.7205/MILMED-D-13-00222 Shultz, R., Anderson, S. C., Matheson, G. O., Marcello, B., & Besier, T. (2013). Test-retest and interrater reliability of the Functional Movement Screen. Journal of Athletic Training, FUNCTIONAL MOVEMENT AND INJURY 51 48(3), 331-336. doi:http://dx.doi.org/10.4085/1062-6050-48.2.11 Smith, C. A., Chimera, N. J., & Warren, M. (2015). Association of y balance test reach asymmetry and injury in Division I athletes. Medicine & Science in Sports & Exercise, 47(1), 136-141. doi:10.1249/MSS.0000000000000380 Smith, P. D., & Hanlon, M. P. (2017). Assessing the effectiveness of the functional movement screen in predicting noncontact injury rates in soccer players. Journal of Strength and Conditioning Research, 31(12), 3327-3332. doi:10.1519/JSC.0000000000001757 Sprague, P. A., Mokha, G. M. & Gatens, D. R. (2014). Changes in Functional Movement Screen scores over a season in collegiate soccer and volleyball athletes. Journal of Strength and Conditioning Research, 28(11), 3155-3163. doi:10.1519/JSC.0000000000000506 Tenan, M. S. (2016). Quantifying emergency department visits from sport and recreation: Focus on the lower extremity and knee, 19972009. Journal of Athletic Training, 51(4), 309316. doi:10.4085/1062-6050-51.4.12 Teyhen, D. S., Riebel, M. A., McArthur, D. R., Savini, M., Jones, M. J., Goffar, S. L., Kiesel, K. B., & Plisky, P. J. (2014). Normative data and the influence of age and gender on power, balance, flexibility, and functional movement in healthy service members. Military Medicine, 179(4), 413-420. doi:https://doi.org/10.7205/MILMED-D-13-00362 Tofthagen, C. (2012). Threats to validity in retrospective studies. Journal of the Advanced Practitioner in Oncology, 3(3), 181-183. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/ Walbright, P. D., Walbright, N., Ojha, H., & Davenport, T. (2017). Validity of functional screening tests to predict lost-time lower quarter injury in a cohort of female collegiate athletes. International Journal of Sports Physical Therapy, 12(6), 948-959. Retrieved FUNCTIONAL MOVEMENT AND INJURY from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5675370/ Warren, M., Smith, C., & Chimera, N. (2015). Association of the functional movement screen with injuries in division I athletes. Journal of Sport Rehabilitation, 24(2), 163-170. doi:10.1123/jsr.2013-0141 Wright, A. A., Dischiavi, S. L., Smoliga, J. M., Taylor, J. B., & Hegedus, E. J. (2017). Association of lower quarter y-balance test with lower extremity injury in NCAA division 1 athletes: An independent validation study. Physiotherapy, 103(2), 231-236. doi:10.1016/j.physio.2016.06.002 52 FUNCTIONAL MOVEMENT AND INJURY 53 Table 1 Comparison of FMS-CS and YBT-LQ Injury Risk Categories by Injury (N = 143) Injured Not injured n = 89 n = 54 n (%) n (%) 2 OR 95% CI p high-risk 38 (63.3%) 22 (36.7%) 0.05 0.92 [0.46, 1.83] .818 low-risk 51 (61.4%) 32 (38.6%) high-risk 27 (62.8%) 16 (37.2%) 0.03 0.94 [0.45, 1.97] .874 low-risk 62 (61.4%) 39 (38.6%) high-risk 30 (51.7%) 28 (48.3%) 4.59 2.12 [1.06, 4.23] .032 low-risk 59 (69.4%) 26 (30.6%) high-risk 35 (60.3%) 23 (39.7%) 0.15 1.15 [0.58, 2.28] .700 low-risk 54 (63.5%) 31 (36.5%) Variable FMS CS YBT Ant YBT Plat YBT Pmed Note. FMS CS = Functional Movement Screen Composite Score; YBT = Y-balance test lower quarter Ant = anterior reach difference 4 cm cut point; Plat = posterolateral reach difference 4 cm cut point; Pmed = posteromedial reach difference 4 cm cut point; Alpha level adjusted to .008 using Bonferroni correction to account for the six hypothesis tests FUNCTIONAL MOVEMENT AND INJURY 54 Table 2 Comparison of Frequency of FMS-CS and YBT-LQ Injury Risk Categories Between Noncontact Injury and No Injury Noncontact Injury No Injury n = 61 n = 54 n (%) n (%) 2 OR 95% CI p high-risk 27 (55.1%) 22 (44.9%) 0.15 0.87 [0.41, 1.82] .703 low-risk 34 (51.5%) 32 (48.5%) high-risk 21 (56.8%) 16 (43.2%) 0.38 0.78 [0.36, 1.72] .538 low-risk 40 (50.6%) 39 (49.4%) high-risk 19 (40.4%) 28 (59.6%) 5.08 2.38 [1.11, 5.09] .024 low-risk 42 (61.8%) 26 (38.3%) high-risk 23 (50.0%) 23 (50.0%) 0.28 1.23 [0.58, 2.59] .593 low-risk 38 (55.1%) 31 (44.9%) Variable FMS CS YBT Ant YBT Plat YBT Pmed Note. FMS CS = Functional Movement Screen Composite Score; YBT = Y-balance test lower quarter Ant = anterior reach difference 4 cm cut point; Plat = posterolateral reach difference 4 cm cut point; Pmed = posteromedial reach difference 4 cm cut point Alpha level adjusted to .008 using Bonferroni correction to account for the 6 hypothesis tests FUNCTIONAL MOVEMENT AND INJURY 55 Table 3 Comparison of FMS-CS scores, FMS Component Test Score Frequency, and YBT-LQ Direction Scores by Gender Males Females n = 71 n = 72 Mdn (IQR) Mdn (IQR) Z value p FMS CS 14 (3) 15 (3) -2.27 .023 YBT Ant 2.30 (3.20) 2.40 (3.50) -0.39 .699 YBT Plat 3.30 (4.70) 3.25 (3.70) -0.28 .782 YBT Pmed 3.10 (4.10) 3.55 (4.50) -0.69 .489 n (%) n (%) 2 p 0.51 .776 3.86 .145 Test FMS Component Test Score Frequency Deep Squat 1 33 (50.8%) 32 (49.2%) 2 35 (50.0%) 35 (50.0%) 3 3 (37.5%) 5 (62.5%) Hurdle Step 1 6 (85.7%) 1 (14.3) 2 58 (47.5%) 64 (52.5%) 3 7 (50.0%) 7 (50.0%) FUNCTIONAL MOVEMENT AND INJURY n (%) 56 n (%) Inline Lunge 1 3 (33.3%) 6 (66.7%) 2 46 (57.5%) 34 (42.5%) 3 22 (40.7%) 32 (50.3%) Shoulder Mobility 1 6 (7.05%) 2 (25.0%) 2 25 (58.1%) 18 (41.9%) 3 40 (43.5%) 52(56.5%) ASLRa 1 12 (92.3%) 1 (7.7%) 2 30 (56.6%) 23 (43.4%) 3 29 (37.7%) 48 (62.3%) Trunk Stab P/Ub 1 11 (22.4%) 38 (77.6%) 2 41 (75.9%) 13 (24.1%) 3 19 (47%) 21 (52.5%) Rotary Stabilityc 1 13 (61.9%) 8 (38.1%) 2 58 (47.5%) 64 (52.5%) 2 p 4.65 .098 4.70 .095 14.91 .001* 29.49 < .001* 1.48 .224 Note. FMS CS = Functional Movement Screen Composite Score; YBT = Y-balance test lower quarter Ant = anterior reach difference 4 cm cut point; Plat = posterolateral reach difference 4 cm FUNCTIONAL MOVEMENT AND INJURY 57 cut point; Pmed = posteromedial reach difference 4 cm cut point; IQR = interquartile range; ASLR = active straight leg raise; Trunk Stab P/U = trunk stability push up *p < .05 a Significant difference (p < .017) between genders on pairwise post-hoc analysis between scores 1 and 3 with the Fisher exact test (p < .001). b Significant difference (p < .017) between genders on all pairwise post-hoc analysis: between scores 1 and 2 2 (1) = 29.39, p <.001, between scores 1 and 3 2 (1) = 6.19, p = .013, between scores 2 and 3 2 (1) = 8.04, p = .005. No counts were less than 5. c No scores of 3 were achieved. FUNCTIONAL MOVEMENT AND INJURY 58 Figure 1. Data Collection Sources YBT-LQ directions (cm) Injury (y/n) Mechanism (contact/noncontact) Lower limb length (cm) FMS scores Gender (male/female) Age, Height, Weight Injury history Electronic Medical Record (ATS) Patient Paper Charts Network Drive FUNCTIONAL MOVEMENT AND INJURY 59 Figure 2. Diagram of Y-Balance Test Lower Quarter Non-Stance Foot Excursions Anterior Anterior Stance Right Stance Left Posterolateral Posteromedial Posteromedial Posterolateral FUNCTIONAL MOVEMENT AND INJURY Appendix A Lebanon Valley College Institutional Review Board Approval 60 FUNCTIONAL MOVEMENT AND INJURY Appendix B Lebanon Valley College and University of Indianapolis Reliance Agreement 61  ...