Juan E. Corrala, Andres J. Yarurb, Liege Diaza, Okeefe L. Simmonsa, Daniel A. Sussmanb
University of Miami Miller School of Medicine, Miami, Florida, USA
aDepartment of Medicine (Juan E. Corral, Liege Diaz, Okeefe L. Simmons), University of Miami Miller School of Medicine, Miami, Florida, USA; bDivision of Gastroenterology (Andres J. Yarur, Daniel A. Sussman), University of Miami Miller School of Medicine, Miami, Florida, USA
Background Patients with inflammatory bowel disease (IBD) require close follow up and frequently utilize healthcare services. We aimed to identify the main reasons that prompted patient calls to gastroenterology providers and further characterize the “frequent callers”.
Methods This retrospective cross-sectional study included all phone calls registered in medical records of IBD patients during 2012. Predictive variables included demographics, psychiatric history, IBD phenotype, disease complications and medical therapies. Primary outcome was the reason for call (symptoms, medication refill, procedures and appointment issues). Secondary outcome was the frequency of changes in management prompted by the call.
Results 209 patients participated in 526 calls. The mean number of calls per patient was 2.5 (range 0-27); 49 (23.4%) patients met the criterion of “frequent caller”. Frequent callers made or received 75.9% of all calls. Crohn’s disease, anxiety, extra-intestinal manifestations and high sedimentation rate were significantly associated with higher call volume. 85.7% of frequent callers had at least one call that prompted a therapeutic intervention, compared to 18.9% of non-frequent callers (P<0.001). The most common interventions were ordering laboratory or imaging studies (15.4%), dose adjustments (12.1%), changes in medication class (8.4%), and expediting clinic visits (8.4%).
Conclusion Most phone calls originated from a minority of patients. Repeated calling by the same patient and new onset of gastrointestinal (GI) and non-GI symptoms were important factors predicting the order of diagnostic modalities or therapeutic changes in care. Triaging calls to IBD healthcare providers for patients more likely to require a change in management may improve healthcare delivery.
Keywords Inflammatory bowel disease, phone calls, high utilization, quality improvement
Ann Gastroenterol 2015; 28 (3): 357-365
Introduction
Patients with inflammatory bowel diseases (IBD) experience frequent symptoms, often requiring regular communication with the healthcare team. IBD patients tend to have higher medical care utilization when compared to those with other gastrointestinal (GI) diseases [1]. Severity of IBD as well as psychosocial factors and the structure of the healthcare system influence their care-seeking behavior [2,3].
For many patients with chronic GI diseases the frequency of scheduled office visits may be inadequate, with the development of acute symptoms or concerns triggering communication behaviors. A small group of these patients, referred to as “frequent callers”, account for the majority of regular and after-hour clinic calls, ranging from 23-51% depending on how “frequent caller” is defined [4,5]. Studies from primary care and gastroenterology clinics show that these “frequent callers” are also high utilizers of in-office visits [6]. These telephone calls can also predict future healthcare service utilization such as future emergency department visits and hospital admissions [5]. For IBD patients, the productivity of these patient-initiated calls in effecting a change in medical management has not previously been described.
This study aims to identify the most frequent concerns prompting patient-initiated calls to his/her healthcare provider and characterize those patients who are frequent callers. Additionally, we assess how often these calls resulted in changes in medical management.
Patients and methods
We designed a retrospective cross-sectional study approved by the University of Miami Miller School of Medicine Institutional Review Board. We included IBD patients 18 years or older followed at the Crohn’s and Colitis Center of the University of Miami (Florida, USA). Patient data were extracted from a database including patients whose index date of care in our healthcare system was between January 1, 2008 and December 31, 2012 [7]. The patients in this database were derived by identifying patients from our electronic medical record (EMR) with the ICD-9-CM diagnoses for Crohn’s Disease and/or ulcerative colitis (555.X and/or 556.X) [7]. Only those who had at least one clinic appointment and complete electronic medical records at our center were included. Approximately two thirds of patients were excluded as they had only the index visit in our system’s hospitals or clinics. All phone calls registered in the EMR during 2012 were reviewed.
Our outpatient clinic is a tertiary referral IBD center. Incoming phone calls are answered by the central GI office staff between the hours of 8 am and 5 pm, and are documented in the EMR. Phone calls initiated by physicians, nurse practitioners, nutrition services, social workers and nurses to patients are recorded in a similar fashion. Phone calls dialed or received by the clinic staff or our system’s answering service after hours (between 5 pm and 8 am) are not routinely registered.
All phone calls (incoming or outgoing) during the study period were reviewed. We collected the following data: the individuals involved on both ends of the conversation, time at which the call occurred, reason for call (e.g. GI symptoms, non-GI symptoms, medication refill, results requests, insurance documents, procedures and appointment issues), and outcomes of the phone call (e.g. medication changes, new laboratory orders, changes in outpatient clinic visits, and referrals for urgent care) were registered.
Predictive variables collected from the EMR included demographic variables (age, gender and ethnicity), IBD phenotype, previous surgery for IBD, medical therapies, insurance status and psychiatric co-morbidities. We included psychiatric history as previous studies suggest that patients with psychiatric disease have higher healthcare utilization [7,8]. IBD phenotype included extra-intestinal manifestations (EIMs), perianal disease, history of surgery and surgical stoma. Patients were considered to have EIMs if they had arthritis, erythema nodosum, pyoderma gangrenosum, aphthous stomatitis and iritis/uveitis as per treating physician.
The most recent serologic biomarkers of inflammation, including C-reactive protein (CRP) and erythrocyte sedimentation rate (ESR) were recorded if available in the first appointment, as they reflect inflammatory activity in IBD [9]. ESR was considered high if greater than 12 mm/h in men, and 18 mm/h in women. CRP was considered elevated if greater than 10 mg/L.
The primary aim was to describe the most common reasons for patient-initiated calls (12 reasons in total): medication refills, new symptoms, old symptoms, appointment issues, insurance issues, reporting new results, specialist referral within same healthcare system, transfer of care to other healthcare centers, new admission to our University hospital, issues arising during hospitalization (in any hospital), clarifications of medications, and “other”. New symptoms were stratified into GI or non-GI symptoms. Reporting new results was considered when the conversation mentioned new laboratory results, new imaging results or new procedure-related reports (e.g. pathology report from colonoscopy). The secondary outcome was the medical decision-making prompted by the call (10 outcomes in total): new prescriptions, change in prescriptions, hospital admissions, appointment changes, referral to other specialties, requesting appointment labs, requesting imaging, requesting procedures, emergency care, and reassurance. Neither the reason for call nor the changes prompted by the call were mutually exclusive.
Patients were considered frequent callers if they were in the higher fourth quartile of phone call frequency. The remaining patients were considered non-frequent callers, including those who did not have any phone call recorded.
Phone calls were classified in two groups depending on whether or not they resulted in a change of management. Changes in treatment included starting a new medication, changes in dose of existing medications, direct admission to the hospital, referral to a healthcare facility (emergency room, outpatient clinics or other hospitals), appointment changes, or request for imaging studies or a special procedure (surgery, endoscopy or tissue biopsy). Three of the authors (JEC, LD, OS) independently analyzed all phone calls recorded in the EMR.
Incoming phone calls where the caller could not be identified in the EMR and outgoing calls where the patient was unavailable (but still were recorded in the EMR) were not included in the analysis. Staff or physician return of a communication initiated by the patient (and vice versa) was considered as one encounter only.
Chi-square and Student’s t-test were used to test differences between nominal and continuous variables, respectively. Univariate logistic regression analysis tested for variables associated with frequent callers and phone calls resulting in treatment changes. After reviewing all available phone calls, 10% of all patients’ charts were selected randomly and were reviewed simultaneously by the three investigators collecting the data to estimate inter-observer agreement. Kappa statistic and bias index were calculated to test inter-observer agreement. Analyses were done using Stata/SE 11.2.
Results
Two hundred and twenty-five patients were screened; 16 were excluded from the analysis due to incomplete EMR leaving 209 patients who met inclusion criteria. The median age of the group was 42 (inter-quartile range 30-58); 105 (50.2%) were male, 53 (25.5%) were Hispanic. 138 (66.0%) had Crohn’s disease, sixty-nine (33.0%) ulcerative colitis, and two (0.9%) indeterminate colitis. The mean time between IBD diagnosis and study enrollment was 14.34±12.02 years. 117 (56%) patients had a recent ESR and 121 (58%) had a recent CRP level (measured in the first visit) (Table 1).
Table 1 General characteristics of non-frequent and frequent callers (n=209)
526 phone calls were recorded during the year 2012. Phone calls per person ranged from 0 to 27, the upper quartile was found to be 4 or more phone calls. 312 (59.3%) were incoming patient-generated calls, 186 (35.3%) were initiated by the provider, and in 28 (5.3%) the call initiator could not be determined. Forty-eight calls (9.1%) involved the patient’s caretaker (i.e. parent or spouse) instead of the patient.
33.8% called to inquire as to study results (20% blood work, 12.9% endoscopic procedures, and 3.1% imaging results), 26.9% concerning GI symptoms (9.0% new symptoms, 17.9% recurrent symptoms), 14.9% for non-GI symptoms, 15.3% for medication refills, 14.2% with questions regarding their medications, 7.7% related with insurance or other documentation issues, and 4.7% had clinic or imaging appointment change requests.
Over two-thirds (n=338) of phone calls occurred in the afternoon (12:00 to 17:00 h), and January was the month with the most phone calls recorded (Fig. 1).
Figure 1 Monthly distribution of incoming and outgoing phone calls
Frequent and non-frequent callers
Ninety-two (44.4%) patients did not have any phone call recorded in their charts. 49 patients (23.4%) were considered frequent callers. Overall, only 11.9% patients were involved in 52.6% of all phone calls (Fig. 2).
Figure 2 Histogram of phone call frequency
Frequent callers were more often women, had Crohn’s disease (versus ulcerative colitis or indeterminate colitis), had a higher prevalence of anxiety, and a lower body mass index (BMI) compared to non-frequent callers. The number of patients who had a family member or another caretaker calling for his/her concerns was similar in both groups. The two markers of inflammation (CRP and ESR) were numerically higher in the frequent callers group, but only ESR reached statistical significance. The baseline characteristics of the frequent callers and non-frequent callers are shown in Table 1.
Regarding IBD therapy and phenotype, ulcerative colitis and monotherapy with aminosalicylates were associated with less frequent calls [odds ratio 0.43 (95%CI 0.20 - 0.93) and 0.36 (0.17-0.74), respectively], while EIMs [3.20 (1.64 - 6.23)] related to more frequent phone calls. Patients’ current age, disease duration (less or more than 10 years), other psychiatric co-morbidities, medications used when enrolling in the study (except for salicylates), and history of previous surgery for IBD did not differ between frequent and non-frequent callers (Table 2).
Table 2 Factors associated with frequent callers (n=209)
Patient-initiated calls
Two-thirds of phone calls were patient-initiated. Thirty-eight (11.8%) of these calls were initiated by a family member or other caretaker. The main reasons for patients calling the clinic are shown in Table 3. Reasons for calling were not exclusive and in 72 (22.3%) phone calls patients raised two, 20 (6.2%) three, 10 (3.1%) four, and 3 (0.9%) six issues to be addressed during the phone encounter.
Table 3 Reasons for patient calling clinic (N=323)
Of these calls, 100 (32.0%) resulted in a change in medical management. 54 (16.7%) ended in the provider ordering labs, 34 (10.3%) in new prescriptions, 28 (8.7%) in an appointment change, 26 (8.1%) in medication adjustments, 7 (2.2%) ordering a procedure (colonoscopy, biopsy), 6 (1.9%) in referral to the ER, 4 (1.24%) requesting the patient to have a planned admission, and 2 (0.6%) requesting imaging studies. Twenty-five incoming calls (7.7%) ended in reassurance of the patient without further intervention.
Phone calls associated with changes in management
Nearly one-third of calls (31.6%) resulted in changes in management. 85.7% of frequent callers had at least one call that prompted a therapeutic change, compared to 18.9% of non-frequent callers (P<0.001). Origin of the phone call (patient vs. provider-initiated) was not associated with the frequency of the provider changing treatment plan (P=0.67). The most common changes were ordering diagnostic laboratory or imaging studies (15.4%), adding medications (12.1%), altering medication dosage/frequency (8.4%), and expediting clinic visits (8.4%).
Calling with either new GI symptoms or non-GI symptoms were strongly associated with changes in management [odds ratio 3.46 (95%CI 2.15-5.55) and 4.28 (2.57-7.11), respectively]. New GI symptoms were 1.74 times more likely than existing GI symptoms to result in a change in medical treatment. Lower BMI, presence of perianal disease, and new findings via imaging were found to be significantly associated with a change in management but the remaining variables were not (Table 4).
Table 4 Factors associated with phone calls ending in change in medical treatment (N=526)
Inter-observer agreement for the three reviewers was 77.1%, 79.3% and 80.5%; Kappa statistics of 0.34, 0.37 and 0.31 and corresponding bias indices of 0.10, 0.07 and 0.05 were calculated. Mean inter-observer agreement was 78.9%, Kappa statistic was 0.34, and mean bias index was 0.07.
Discussion
We identified several epidemiologic and clinical variables of IBD patients that were associated with a high number of calls to a tertiary level GI clinic. Markers of disease activity (elevated ESR and lower BMI), the presence of EIMs, and anxiety were associated with being in the high-frequency caller category. Calling with any symptoms (either GI or non-GI) and new findings with imaging studies were the features most associated with changes in medical management by phone.
In U.S. tertiary IBD centers, common reasons for calling are follow up of patients’ active medical problems, refill requests, insurance authorizations, form completion, and record requests [5]. In this scenario, patient concerns are better addressed through triage of phone calls by office staff, with the direction of most calls to clerical providers with the tools and skills needed to obtain insurance authorizations, schedule diagnostic tests, and refill chronic medications. Other technologies like voice-activated response systems have been shown to improve staff utilization in large volume call centers (e.g. poison control hotlines) [10]. Of all calls, less than half are due to active medical symptoms and only a small percentage result in physicians requesting new diagnostic studies or treatment changes [5]. Physicians often complain that responsibilities like answering phone calls and emails, insurance/billing paperwork and other documentation obligations are excessive, and compromise their time with patients [11]. Reducing the time the IBD care providers devote to phone calls can provide additional time for clinical responsibilities. This is particularly relevant in IBD patients who have higher healthcare utilization, require specialty medications that demand more paperwork, and are at higher risk of poly-pharmacy [12,13].
Our findings agree with previous studies that identify IBD high-utilizers to be generally female with psychological co-morbidities [6]. Likewise, patients with Crohn’s disease receiving corticosteroids that have high CRP and ESR levels have been found to be frequent callers in other tertiary clinical centers [5]. In our study, patients treated only with salicylates called less frequently than those on other treatments, a finding likely explained by the ready response to these medications with the mild-to-moderate phenotype of ulcerative colitis and the less severe symptoms experienced by these patients [5,6]. Similarly, underweight (but not normal weight, overweight or obesity) was associated with higher frequency of calls, suggesting that chronically severe disease can increase the frequency of phone calls. Other variables found to be strongly associated with phone call activity in other studies are quality of life (measured by The Short Inflammatory Bowel Disease Questionnaire) and chronic abdominal pain [5]. These variables were not addressed in this study given our retrospective design, but should be considered when designing tools to triage outpatient phone calls.
In our population, frequent callers were more likely to be involved in a phone call that prompted a change in the management or a new order for diagnostic studies. This finding needs to be interpreted cautiously as repeated observations inherently increase the likelihood of identifying an endpoint. However, this high rate in treatment/diagnostic changes may also be explained by a higher frequency of active, symptom-generating disease in these individuals.
South Florida is a unique environment, with a large Hispanic and Caribbean population and high rate of patient transiency due to migration patterns. This ethnic diversity entails different cultural norms to seek health care and to communicate with healthcare providers (e.g. patients with language barriers might avoid phone consultation). We did not find a difference between Hispanics and non-Hispanics regarding phone call frequency. A study done in California also showed that IBD healthcare utilization does not vary significantly between Hispanics, non-Hispanic whites, blacks and Asians when they have similar access to care after adjusting for disease severity [3]. We found that one of 30 calls recorded was to request records be transferred to other medical centers in or out of state. The monthly trends in phone calls might be influenced by migratory patterns and we hypothesize that the spike of phone calls seen in January is related to the influx of “snowbirds” and travelers to Florida in winter [14].
Despite the explosion of health information resources through broadcast and internet-based media, patients still rank their personal physicians as their most desired source of information (68-73% of irritable bowel syndrome patients did so in two different studies) [15,16]. Cultural norms, busy schedules and physicians’ preferences determine access to direct communication [16,17]. While patients would prefer having direct access to their physicians, this raises several concerns due not only to demands on physician time but also the lack of remuneration for services when responding to patient calls. In Australia, novel strategies are being studied to track and bill phone calls and electronic mailing, dependent on the duration of communication [18]. In our study, only a limited percentage of calls required immediate clinical decisions, and most phone calls could have been managed by support personnel, without needing assessment or billing by the physician.
The reason for call (new or old symptoms, in particular) was more relevant than all other patient characteristics including patient demographics, IBD phenotype (other than presence of perianal disease), nutritional or psychiatric conditions. These variables can be assessed in seconds, and do not require intensive chart review by office staff. Having an electronic system in place whereby a change in symptom pattern triggers an expedited office visit or rapid contact with a physician extender is a desirable intervention. Appropriate triage of calls may provide faster resolution of patient concerns, increase patient satisfaction and serve as a resource saving strategy.
This study has several strengths and limitations. Even though previous studies with more statistical power have already described frequent callers, this is the first study to establish what type of phone calls lead to a change in management [5]. This provides evidence to support initiatives to triage phone calls based on reason for call. Also, inter-observer agreement for acquisition of data points was acceptable but kappa statistic was only fair [19]. We attribute the suboptimal kappa level to the fact that reviewers were trained independently. Bias index was low suggesting that discordances were symmetrical [20,21]. Further subgroup analyses and other statistical analyses (i.e. multivariate regression) were not performed due to the limited sample size [22].
One of the larger limitations of this study was that call encounters not recorded in the chart would have been missed, and the frequency of these events cannot be tracked. Accordingly, there is no record of email communications between patients and their providers. Some centers discourage email communications between patients and providers given concerns for violations in privacy laws. Because of the cross-sectional nature of this study, no information regarding how the disease course was affected by phone consultation could be reported. Also, patients’ preferred method of communication with their healthcare providers should also be considered in future studies (i.e. text messages, email), as phone calls may not be the first choice for a relatively young, tech-savvy population. Such alternatives might be particularly appealing for those patients who never called (44.4%) but may have had concerns or questions. Finally, this study was limited to calls done during regular working hours, which likely underestimates a subset of the population who call after-hours for acute events that require prompt care, diagnostic studies, and likely more ER visits and hospital admissions.
Our center cares for a significant number of IBD patients, with outside-of-clinic communications by phone contributing to a large allocation of provider and ancillary resources. The stringent exclusion criteria we invoked in the study design requiring complete medical records consisting of more than one clinical encounter, laboratory results, and the assessment of presence/absence of all variables substantially diminished our sample size, potentially limiting the generalizability of our results. Also, our tertiary center cares for a disproportionately high percentage of patients with moderate-to-severe disease when compared to a general gastroenterology practice and the frequency of calls may not apply in the rest of the community. Our study identifies patient characteristics that would predict the need for patient-directed communication resulting in clinical management changes outside of the traditional face-to-face clinical encounter. Tracking the nature, duration, and complexity of these calls is of utmost importance given the need to demonstrate to third-party payers the quality and quantity of difficult decision-making that is generated outside of the direct clinic space. Many groups have supported reimbursing these encounters, and EMRs offer the opportunity to track these events with great efficiency [18]. These data also emphasize the importance of novel technologies like mobile phone applications to assist patients with self-monitoring symptoms and communicating with healthcare providers [23-26].
In conclusion, many patient-initiated calls resulted in a change in medical management. Repeat calling by the same patient and the new onset of GI and non-GI symptoms were important factors predicting the order of diagnostic modalities or therapeutic changes in care. Triaging of calls to IBD healthcare providers for those patients more likely to require a change in management may improve health outcomes, patient satisfaction and should be considered as a quality improvement measure for GI practices.
What is already known:
Patients with inflammatory bowel disease tend to have higher medical care utilization compared to those with other gastrointestinal (GI) diseases
Frequent callers are also high utilizers of in-office visits, emergency department visits, and hospital admissions
Few studies have described frequent callers and studies analyzing what type of phone calls lead to a change in disease management have been missing
What the new findings are:
Frequent callers were more commonly female, with psychological co-morbidities, had Crohn’s disease, and elevated markers of inflammation (C-reactive protein and erythrocyte sedimentation rate)
Frequent callers were more commonly female, with psychological co-morbidities, had Crohn’s disease, and elevated markers of inflammation (C-reactive protein and erythrocyte sedimentation rate)
Triaging of calls to healthcare providers for those patients more likely to require a change in management may improve health outcomes and patient satisfaction
References
1. Verhoef MJ, Sutherland LR, Outpatient health care utilization of patients with inflammatory bowel diseaseDig Dis Sci 1990; 35: 1276-1280.
2. Deter HC, von Wietersheim J, Jantschek G, Burgdorf F, Blum B, Keller W, High-utilizing Crohn’s disease patients under psychosomatic therapyBiopsychosoc Med 2008; 2: 18.
3. Li D, Collins B, Velayos FS, Liu L, Lewis JD, Allison JE, Flowers NT, Hutfless S, Abramson O, Herrinton LJ, Racial and ethnic differences in health care utilization and outcomes among ulcerative colitis patients in an integrated health-care organizationDig Dis Sci 2014; 59: 287-294.
4. Hildebrandt DE, Westfall JM, Reasons for after-hours callsJ Fam Pract 2002; 51: 567-569.
5. Ramos-Rivers C, Regueiro M, Vargas EJ, Szigethy E, Schoen RE, Dunn M, Watson AR, Schwartz M, Swoger J, Baidoo L, Barrie A, Dudekula A, Youk AO, Binion DG, Association Between Telephone Activity and Features of Patients with Inflammatory Bowel DiseaseClin Gastroenterol Hepatol 2014; 12: 8.
6. Hildebrandt DE, Westfall JM, Nicholas RA, Smith PC, Stern J, Are frequent callers to family physicians high utilizers?Ann Fam Med 2004; 2: 546-548.
7. Panara AJ, Yarur AJ, Rieders B, Proksell S, Deshpande AR, Abreu MT, Sussman DA, The incidence and risk factors for developing depression after being diagnosed with inflammatory bowel disease: a cohort studyAliment Pharmacol Ther 2014; 39: 802-810.
8. Bennebroek Evertsz F, Bockting CL, Stokkers PC, Hinnen C, Sanderman R, Sprangers MA, The effectiveness of cognitive behavioral therapy on the quality of life of patients with inflammatory bowel disease: multi-center design and study protocol (KL!C- study)BMC Psychiatry 2012; 12: 227.
9. Solem CA, Loftus EV, JrTremaine WJ, Harmsen WS, Zinsmeister AR, Sandborn WJ, Correlation of C-reactive protein with clinical, endoscopic, histologic, and radiographic activity in inflammatory bowel diseaseInflamm Bowel Dis 2005; 11: 707-712.
10. Krenzelok EP, Mrvos R, The use of an automated interactive voice response system to manage medication identification calls to a poison centerClin Toxicol (Phila)2009; 47: 425-429.
11. Christino MA, Matson AP, Fischer SA, Reinert SE, Digiovanni CW, Fadale PD, Paperwork versus patient care: a nationwide survey of residents’ perceptions of clinical documentation requirements and patient careJ Grad Med Educ 2013; 5: 600-604.
12. Buckley JP, Kappelman MD, Allen JK, Van Meter SA, Cook SF, The burden of comedication among patients with inflammatory bowel diseaseInflamm Bowel Dis 2013; 19: 2725-2736.
13. Gleason PP, Alexander GC, Starner CI, Health plan utilization and costs of specialty drugs within 4 chronic conditionsJ Manag Care Pharm 2013; 19: 542-548.
14. Smith SK, House M, Snowbirds, sunbirds, and stayers: seasonal migration of elderly adults in FloridaJ Gerontol B Psychol Sci Soc Sci 2006; 61: S232-S239.
15. Halpert A, Dalton CB, Palsson O, Patient educational media preferences for information about irritable bowel syndrome (IBS)Dig Dis Sci 2008; 53: 3184-3190.
16. Drossman DA, Morris CB, Schneck S, International survey of patients with IBS: symptom features and their severity, health status, treatments, and risk taking to achieve clinical benefitJ Clin Gastroenterol 2009; 43: 541-550.
17. Wong RK, Tan JS, Drossman DA, Here’s my phone number, don’t call me: physician accessibility in the cell phone and e-mail eraDig Dis Sci 2010; 55: 662-667.
18. Rutland J, Marie C, Rutland B, A system for telephone and secure email consultations, with automatic billingJ Telemed Telecare 2004; 10 : Suppl 188-91.
19. Viera AJ, Garrett JM, Understanding interobserver agreement: the kappa statisticFam Med 2005; 37: 360-363.
20. Ludbrook J, Detecting systematic bias between two ratersClin Exp Pharmacol Physiol 2004; 31: 113-115.
21. Sim J, Wright CC, The Kappa statistic in reliability studies: use, interpretation, and sample size requirementsPhys Ther 2005; 85: 257-268.
22. Katz MH, Multivariable analysis: a primer for readers of medical researchAnn Intern Med 2003; 138: 644-650.
23. Bengtsson U, Kasperowski D, Ring L, Kjellgren K, Developing an interactive mobile phone self-report system for self-management of hypertension. Part 1:Patient and professional perspectivesBlood Press 2014;
24. Licskai C, Sands TW, Ferrone M, Development and pilot testing of a mobile health solution for asthma self-management: asthma action plan smartphone application pilot studyCan Respir J 2013; 20: 301-306.
25. Atreja A, Rizk M, Capturing patient reported outcomes and quality of life in routine clinical practice: ready for prime time?Minerva Gastroenterol Dietol 2012; 58: 19-24.
26. Brennan PF, Personal health technology can improve outcomesMod Healthc 2014; 44: 27.
Notes
Conflict of Interest: Daniel A. Sussman has served in a medical advisory position to ExactSciences. He also has research funding from Bankhead-Coley Team Science (2BT02), Olympus America Inc., American Cancer Society (IRG-98-277-13), and Sylvester Comprehensive Cancer Center. None of these organizations participated in the study design, data collection or analysis, or in the writing of the manuscript