ABSTRACT: Tracking healthcare-associated infections (HAIs) is crucial for reducing and preventing transmission. This study aimed to evaluate the validity and reliability of the Canadian Nosocomial Infection Surveillance Program (CNISP) COVID-19 surveillance data by assessing key metrics, including case definition, case classification, and outcomes.
Erin McGill, MSc¹*, Joelle Cayen, BHSc¹, Jennifer Ellison, MSc², Diane Lee, MPH¹, Linda Pelude, MSc¹, Robyn Mitchell, MHSc¹, Charles Frenette, MD³, Nisha Thampi, MD, MSc, FRCPC4, Kathryn Bush, MSc²
¹ Canadian Nosocomial Infection Surveillance Program, Public Health Agency of Canada, Ottawa, Ontario, Canada
² Alberta Health Services, Calgary, Alberta, Canada
³ McGill University Health Centre, Montreal, Quebec, Canada
4 Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada
*Corresponding author:
Erin McGill
130 Colonnade Rd.
Ottawa, ON, K1A 0K9
Canada
Email: erin.mcgill@phac-aspc.gc.ca
ABSTRACT
Background: Tracking healthcare-associated infections (HAIs) is crucial for reducing and preventing transmission. This study aimed to evaluate the validity and reliability
of the Canadian Nosocomial Infection Surveillance Program (CNISP) COVID-19 surveillance data by assessing key metrics, including case definition, case classification,
and outcomes.
Methods: In December 2022, a survey containing 12 COVID-19 case study questions was administered to staff from 81 eligible hospitals across 32 hospital networks. These staff members were responsible for submitting data using a standardized protocol and case definitions.
Results: Fifty-four (67%) of the 81 CNISP hospital sites completed the survey. The mean survey score was 79% with a median of 83%, and a range of 58-91%. Scores varied by question theme, from 70% for reasons for admission, to 93% for multiple positives.
Conclusion: The study findings indicate that CNISP case definitions and classifications were consistently and accurately applied across most case study questions.
These results underscore the robust quality of COVID-19 data gathered through the national surveillance platform.
KEYWORDS
Surveillance, COVID-19, public health, data quality, competency
INTRODUCTION
The Canadian Nosocomial Infection Surveillance Program (CNISP) is a longstanding collaboration between the Public Health Agency of Canada (PHAC), including the National Microbiology Laboratory (NML), the Association of Medical Microbiology and Infectious Diseases Canada (AMMI Canada) and acute care hospitals across Canada (PHAC, 2022). This sentinel surveillance system actively monitors and reports healthcare-associated infections (HAIs) and antibiotic-resistant organisms (AROs) in Canadian acute care facilities. CNISP aims to facilitate the prevention, control, and reduction of HAIs and AROs using data from participating hospitals to measure infection burden, establish benchmark rates for internal and external comparison, identify risk factors, and facilitate the assessment of infection prevention and control (IPAC) measures to improve patient care quality.
Surveillance protocols for CNISP include standardized methods and case definitions (CNISP, 2023). Trained IPAC staff adhere to CNISP surveillance protocols to identify patients eligible for inclusion. Hospitals outside of the CNISP network can compare their internal performance against national CNISP trends and apply CNISP methods and definitions to ensure valid comparisons. While these protocols and definitions aim to maximize consistency across hospitals, standardized surveillance case definitions and protocols may not address every potential patient scenario (Write et al., 2010).
CNISP conducts surveillance of viral respiratory infections (VRIs) where hospitals submit patient-level questionnaire data on a quarterly basis. CNISP began collecting weekly aggregate of VRI data after the emergence of SARS-CoV-2 in early 2020, including among 81 acute care hospitals in 10 provinces and one territory, and quarterly patient-level questionnaires. Despite previous reliability audits of surveillance data for other infections (Forrester et al., 2012; Leduc et al., 2015; Ellison et al., 2023), a specific reliability audit for COVID-19 infections has not yet been conducted. Given the novel nature of SARS-CoV-2 and evolving practices in testing and clinical management since its first detection in Canada on January 25, 2020 (Sunnybrook Hospital, 2020), ongoing evaluation is crucial.
The CNISP network conducted a case study to evaluate the accuracy of applying the COVID-19 case surveillance definition and other critical metrics. This paper seeks to assess the validity and reliability of the CNISP surveillance definition for SARS-CoV-2 infections through assessment of individual case studies.
METHODS
Subject matter experts, including members of the CNISP Data Quality Working Group and VRI Working Group developed and validated case studies (see Appendix I in online edition) for inclusion in an online survey. The survey was conducted using Voxco, an electronic data collection survey software, and included 12 multiple-choice questions. These case studies aimed to assess staff proficiency in applying COVID-19 case surveillance definitions, including eligibility for inclusion, identify multiple positives, application of acquisition classification (healthcare-acquired – your facility, healthcare-acquired – another facility or community-associated), assessing reasons for admission (due to COVID-19 or incidental COVID-19 admission), reporting treatment options and determining 30-day outcomes. All responses were kept confidential.
Given the potential for multiple COVID-19 tests per patient, data collectors were instructed to use the date of the first positive test within the past 90 days to determine eligibility.
Staff from 81 participating CNISP hospitals, across 32 hospital networks were invited to complete the case study survey between December14,2022 and January13,2023. CNISP categorizes hospital networks into three regions:
East (New Brunswick, Newfoundland and Labrador, Nova Scotia, and Prince Edward Island), Central (Ontario, Quebec and Nunavut), and West (Alberta, British Columbia, Manitoba, Saskatchewan). As this was a data quality investigation, each site could have multiple responders, and staff had the option to respond as a group or individually. In this study, 81 CNISP hospitals were considered the denominator for the response rate, and the total number of responses served as the denominator for question scores.
The survey results were exported from Voxco and subsequently cleaned and analyzed using R version 4.2.3
(R Core Team,2023). Entries lacking a valid hospital identifier were excluded, as were responses from hospital staff who opened the survey, but did not respond to any questions. Scores for the 12 questions were evaluated individually
and aggregated by theme (acquisition, antimicrobials, eligibility, multiple positives, outcomes, reason for admission, treatment). Mean, median and range of scores for correctly applying COVID-19 case surveillance definitions were calculated based on the number of responses. A t-test was used to assess statistical significance between group and individual responses.
The survey was either deemed exempt from ethics approval requirements, or received approval from the research ethics board at participating CNISP hospitals, depending on institution-specific policies (Government of Canada, 2018).
RESULTS
A total of 30 responses were received, representing 54 out of 81 (67%) participating CNISP hospitals. It’s important to note that a single response could represent multiple hospitals within a hospital network. Among the 30 responses, 20 (66%) were individual respondents, and 10 (33%) were group respondents. Response rates were highest in the Western region (Table 1).
The mean score for correctly applying COVID-19 case surveillance definitions across the entire survey was 79% (284 out of 360 case study questions), with a median of 83% and a range from 58-91%. Scores varied by question theme, with the highest score (93% or 28 out of 30) for questions related to correctly attributing the first positive of multiple tests (i.e., multiple positives), and the lowest score (70% or 63 out of 90) for questions attributing the reason for admission (Table 2).
There was no significant difference observed between responses from groups compared to individuals, with scores of 80% and 78%, respectively (p = 0.6).
DISCUSSION
We assessed the validity and reliability in the application of CNISP COVID-19 surveillance definitions and found that overall, 79% of case study responses correctly and consistently applied the case definitions, case classifications, and management/treatment criteria for COVID-19 cases. Reliable and valid data are crucial for a national sentinel surveillance system that provides benchmark rates for hospitals across Canada. Results from the case studies can help identify which sections of the VRI protocol, including case surveillance definitions, may require additional clarification or the development of specific tools.
The lowest-scoring theme (70%) was reason for admission. This low score likely reflects challenges in ascertaining the reason for a patient’s admission during a pandemic, which should be considered when interpreting the data. Other research corroborates the difficulties associated with determining the reason for admission during the pandemic, with incidental SARS-CoV-2 detected in 12-26% of hospitalizations (Tsai et al., 2021; Klann et al., 2022). In our study, participants were able to provide comments at the end of the survey. One of the three questions under the theme “reason for admission” was identified by several participants as particularly challenging. The case studies used in the survey were designed to test staff’s ability to apply the definitions, with examples drawn from real patient scenarios which required clarification from CNISP staff. This complexity could have contributed to the lower score. Developing an algorithm to ascertain the reason for admission, similar to the one developed following the C. difficile case study, may improve the validity and reliability in the application of this definition (CNISP, 2019). However, the classification is ultimately subject to clinical judgement, which can vary between infection control professionals (ICPs).
Although hospitals submitting responses as a group had a slightly higher average (80%) than those from individuals (78%), the difference was insignificant. This suggests that collaboration between ICPs can enhance data quality, while also demonstrating that staff are individually well-positioned to apply the CNISP surveillance definitions effectively.
CNISP has previously conducted reliability audits for various surveillance programs, including methicillin-resistant Staphylococcus aureus (MRSA) and vancomycin-resistant enterococci (VRE). The results suggest the data are reliable. For MRSA, the percentage of discordant responses between hospital data and case forms ranged from 3.5% (sex) to 23.7% (acquisition source) (Forrester et al., 2012). For VRE, the percentage of discordant responses ranged from 1% (type of infection) to 22% (previous hospitalization) (Leduc et al., 2015). In 2019, CNISP conducted a case study to assess the validity and reliability of bloodstream infection (BSI) surveillance definitions, finding that 88% of responses correctly applied the surveillance definitions (Ellison et al., 2023). SARS-CoV-2 is a novel pathogen, and although 79% of responses accurately applied the surveillance definitions, the mean score was lower than surveillance programs for more established pathogens, such as MRSA (Forrester et al., 2012; Leduc et al., 2015; Ellison et al., 2023). Several factors might explain the lower mean score for COVID-19 surveillance definitions, including the complexity of the definitions themselves, the availability of data in patient charts, complex patient scenarios, and the greater need for judgement in these cases.
There were several strengths to the study, including its high response rate and overall high scores for the case study questions. The response rate was 67% (54 out of 81 participating sites), surpassing the research goal of 60% (Fincham, 2008; Lindemann, 2021), and exceeding the response rates of previous CNISP case studies such as the BSI case study (58%) (Ellison et al., 2023) and C. difficile case study (53%) (CNISP, 2019). A high response rate during a pandemic illustrates the CNISP network’s strong engagement in the program. Other surveillance networks, including the National Healthcare Safety Network (NHSN) of the Center for Disease Control (CDC), the Texas Department of State Health Services and the Canadian Primary Care Sentinel Surveillance Network (CPCSSN), have conducted similar case study investigations validating the application of surveillance definitions and criteria (Wright et al., 2017; Williamson et al., 2014; Adams et al., 2022). In the NHSN investigation, correct responses were selected 62.5% of the time across 22 case studies (Adams et al., 2022).
In the CPCSSN study, accuracy in applying case definitions ranged from 78% to 99% (Williamson et al., 2014). Only 6%
(5 out of 88) of participants in the Texas Department investigation correctly identified all elements in both case scenarios (Adams et al., 2022). While the results from the CNISP COVID-19 case study fared better than the NHSN or Texas investigations, they were similar to the CPCSSN investigation.
There were several limitations to this case study. Only half of the eastern sites were represented, which may result in the findings not being fully representative of eastern hospitals. Further, the VRI protocol underwent numerous revisions to reflect the evolving understanding of the pandemic, including changes to the case definition, the implementation of new practices, and shifting priorities in data collection. To address this limitation, respondents were instructed to follow the 2022 protocol, however, the survey may have suffered from selection bias, as those most willing to complete the survey are likely those most familiar with CNISP surveillance and applying the definitions. The survey included only 12 case studies to limit the time required for completion. Despite the experience level of staff, it is still possible for cases to be miscoded due to their complexity (Holmes et al., 2022). While staff experience was not assessed, future surveys could examine whether ICPs’ experience and familiarity with CNISP protocols influenced response rates and case study scores by analyzing the information according to CIC certification or number of years participating in CNISP surveillance. A consideration for future CNISP case studies could involve adult- and pediatric-specific case study questions to ensure the validity of data.
This was the third case study conducted by CNISP to assess the validity and reliability of applying surveillance definitions (Ellison et al., 2022). The results of the case study are instrumental in interpreting CNISP COVID-19 surveillance data, guiding enhancements for future protocols, and identifying the need for additional tools. This study underscored the quality of COVID-19 surveillance data collected through CNISP.
The network plans to continue conducting case studies to assess data quality and will focus on the other surveillance modules such as carbapenemase-producing organisms,
central-line associated bloodstream infections and surgical
site infections.
REFERENCES
Adams, J., Mauldin, T., Yates, K., et al. (2022). Factors related to the accurate application of NHSN surveillance definitions for CAUTI and CLABSI in Texas hospitals: A cross-sectional survey. American Journal of Infection Control, 50(1), 111-113.
https://doi:10.1016/j.ajic.2021.07.007
CNISP protocols & publications. IPAC Canada. (2023). https://ipac-canada.org/cnisp-publications Retrieved June 2023.
Canadian Nosocomial Infection Surveillance Program (CNISP) (2019). CDI Data Quality Review. Presented at the 2019 CNISP annual meeting. Unpublished presentation.
Ellison J., Cayen J., Pelude L., Mitchell R., Bush K. (2023) Evaluation of the accuracy in the application of the Canadian Nosocomial Infection Surveillance Program (CNISP) bloodstream infection surveillance definitions. Canadian Journal of Infection Control, 38(41), 19-22. Evaluation_of_the_accuracy_in_the_application.pdf (cjic.ca)
Fincham J.E. Response rates and responsiveness for surveys, standards, and the Journal. American Journal of Pharmaceutical Education. 2008 Apr 15;72(2):43. doi: 10.5688/aj720243. PMID: 18483608; PMCID: PMC2384218.
Forrester, L., Collet, J.C., Mitchell, R., et al. (2012). How reliable are national surveillance data? Findings from an audit of Canadian methicillin-resistant Staphylococcus aureus surveillance data. American Journal of Infection Control, 40(2),102-107. https://doi:10.1016/j.ajic.2011.03.005
Government of Canada (2018). Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2. https://ethics.gc.ca/eng/policy-politique_ tcps2-eptc2_2018.html
Holmes, K., Moinuddin, M., Steinfeld, S. (2022). Developing valid test bank of surveillance case study scenarios for inter-rater reliability. American Journal of Infection Control, 50(8), 960-962.
Klann J, Strasser Z, Hutch M, Kennedy C, Marwaha J, Morris M, Samayamuthu M, Pfaff A, Estiri H, South A, Weber G, Yuan W, Avillach P, Wagholikar K, Luo Y, The Consortium for Clinical Characterization of COVID-19 by EHR (4CE), Omenn G, Visweswaran S, Holmes J, Xia Z, Brat G, Murphy S. Distinguishing Admissions Specifically for COVID-19 from Incidental SARS-CoV-2 Admissions: National Retrospective Electronic Health Record Study. Journal of Medical Internet Research 2022;24(5):e37931 URL: https://www.jmir.org/2022/5/e37931 DOI: 10.2196/37931
Leduc, S., Bush, K., Campbell, J., et al. (2015). What can an audit of national surveillance data tell us? Findings from an audit of Canadian vancomycin-resistant enterococci surveillance data. Canadian Journal of Infection Control, 30,75-81.
Lindemann, N. (2024, January 10). What’s the average survey response rate? [2021 benchmark]. Pointerpro. https://pointerpro.com/blog/average-survey-response-rate
Public Health Agency of Canada (PHAC) (2022, December 13). The Canadian nosocomial infection surveillance program (CNISP). Public Health Agency of Canada. https://health-infobase.canada.ca/cnisp/index.html Retrieved June 2023
R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org
Sunnybrook Hospital. A look back at Canada’s first COVID-19 case. Toronto, ON: SBH; Aug 25, 2020. https://sunnybrook.ca/media/item.asp?page=38&i=2167
Tsai J, Traub E, Aoki K, Oyong K, Sato H, Rizik-Baer D, Gounder P. Incidentally Detected SARS-COV-2 Among Hospitalized Patients in Los Angeles County, August to October 2020. Journal Hospital Medicine. 2021 Aug;16(8):480-483. doi: 10.12788/jhm.3641. PMID: 34328848; PMCID: PMC8340959.
Williamson T., Green M.E., Birtwhistle R., Khan S., Garies S., Wong S.T., Natarajan N., Manca D., Drummond N. Validating the 8 CPCSSN case definitions for chronic disease surveillance in a primary care database of electronic health records. Annals Family Medicine. 2014 Jul;12(4):367-72. doi: 10.1370/afm.1644. PMID: 25024246; PMCID: PMC4096475.
Wright, M.O., Hebden, J.N., Allen-Bridson, K., Morrell, G.C., Horan, T. (2010). Healthcare-associated Infections Studies Project: An American Journal of Infection Control and National Healthcare Safety Network Data Quality Collaboration. American Journal of
Infection Control, 38(5), 416-418. https://doi:10.1016/j.ajic.2010.04.198
Wright, M.O., Allen-Bridson, K., Hebden, J.N. (2017). Assessment of the accuracy and consistency in the application of standardized surveillance definitions: A summary of the American Journal of Infection Control and National Healthcare Safety Network case studies, 20102016. American Journal of Infection Control, 45(6), 607-611. https://doi:10.1016/j.ajic.2017.03.035