Abstract
Background:
Clinical guidelines (CGs) need to be updated to ensure the ongoing validity of recommendations.
Aims and objectives:
This systematic review identified and described the most recent CG update processes, including prioritisation methods, used by international or national groups who provide methodological guidance for developing and updating CGs.
Methods:
Methodological handbooks were identified by searching a predefined list of national and international organisations, and by grey literature searching. A systematic literature search (2011–2021) of Medline, Embase and the Cochrane Library was conducted to identify peer-reviewed articles that described the development and or evaluation of update processes. Data were extracted by one reviewer and checked by a second. Quality assessment was conducted independently by two reviewers. A narrative synthesis was undertaken.
Findings:
In total, 16 handbooks from 11 organisations and three peer-reviewed articles were included. Few handbooks provided comprehensive details beyond whether an update was indicated, with processes for prioritisation of updates and required resources generally lacking; terminology and definitions differed across organisations. In general, evidence synthesis methods used to update CGs were the same as those used to develop CGs de novo.
Discussion and conclusion:
Updating CGs is critical to support policy and practice. It is an iterative process that is both resource-intensive and time-consuming. International or national groups who provide methodological guidance for developing and updating CGs should consider providing more comprehensive guidance and standardising the terminology used to facilitate optimal updating of CGs and prioritisation of CGs for updating.
Background
Clinical guidelines (CGs) are systematically developed statements, based on a thorough evaluation of the evidence, to assist practitioners’ and service users’ decisions about appropriate healthcare for specific clinical circumstances across the entire clinical system (NCEC, 2019). The recommendations contained within CGs are primarily underpinned by evidence syntheses, for example, systematic reviews or adaptation of existing CGs and or recommendations (Sharp et al, 2021). Ongoing evolution of the scientific literature brings the emergence of new evidence which can change the findings of a systematic review and, as a consequence, change the recommendations made within a CG. As such, CGs need to be updated regularly to ensure the validity of the recommendations contained within (Vernooij et al, 2014). Updating CGs is an iterative process that is both resource-intensive and time-consuming. Typically, CGs are updated in accordance with a predefined time period. However, it is also acknowledged that deciding to update a CG depends on other factors, such as the volume of new research published on the topic, clinical burden of the topic, economic impact and resources available to update a guideline (NCEC, 2019). Evidence-based policymaking and evidence-based medicine are the cornerstone of public health and clinical care. To support evidence-based policymaking in this context, there is a need to ensure CGs are based on the most up-to-date evidence. As such, the methodologies for updating CGs, and reporting of updated CGs, should be standardised (Clyne et al, 2023). For that reason, policymakers and other stakeholders are advocating for a move away from updating guidelines based on a predefined time period and moving towards updating guidelines based on prioritisation criteria, to ensure appropriate use of resources (Martinez Garcia et al, 2017). Just as evolution of the scientific literature brings new clinical evidence that can impact the recommendations within a CG, it also brings advancement in methodologies used in development and updating of CGs (Vernooij et al, 2014). Therefore, the purpose of this systematic review was to describe the current guideline update processes, including up-to-date prioritisation methods, used by international or national groups who provide methods guidance for developing and updating CGs. The focus of this systematic review was not on adaptation, adoption, or development of guidelines de novo, but on processes by which existing guidelines are updated. By describing the CG updating and prioritisation methodologies used by internationally recognised organisations, this systematic review will support those responsible for using, developing and updating CGs, and those responsible for developing the methodologies used in CG development and updating.
Methods
In reporting this systematic review we have adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria (Page et al, 2021). The protocol for this systematic review was registered on PROSPERO (registration number: CRD42021274400) and published in HRB Open (Cardwell et al, 2021).
Information sources and search strategy
Due to changes in process and methodologies in guideline development in the last 10 years, the overall search span for this review was the last 10 years (2011–2021). Through scoping searches, we identified two published systematic reviews; one (Vernooij et al, 2014) was a systematic review of methodological handbooks (search date 2013) that provide guidance for updating clinical practice guidelines, and the other (Martinez Garcia, 2017) was a systematic review of peer-reviewed articles (search date 2016) that describe prioritisation processes for updating guidelines. These systematic reviews were considered index documents to check that our search strategy had identified the expected literature.
The websites of 17 organisations (Supplementary file 1) were searched for relevant methodological handbooks. The organisations were chosen based on identification of the organisations from previous systematic reviews on this topic, advice from commissioners and appraisers of CGs in the Department of Health in Ireland, and guidance being available in English. All organisations were contacted (via email) to fill any information gaps identified. Other sources of grey literature searched for relevant methodological handbooks were Google (first 10 pages) and reference chasing; the latter involved searching the reference lists of both index documents and using Google Scholar to scan who had cited the index documents (forward citation searching).
Three databases (Medline (EBSCO), Embase (OVID) and the Cochrane Library) were searched for peer-reviewed articles using the search strategy defined in Supplementary file 2. These databases were chosen in accordance with the two previous index systematic reviews (Vernooij et al, 2014; Martinez Garcia et al, 2017).
Screening
The review question was formulated in line with a modified version of the CIMO (Context, Intervention, Mechanism, Outcome) framework (Denyer et al, 2008). This modified version of the framework (which consisted of the elements, Context, Intervention and Outcome) was used to frame the eligibility criteria and screen publications for inclusion in this review.
Eligibility criteria
Context
- •The publication refers to clinical guidelines that require updating to maintain relevancy.
Intervention
- •The publication was produced by and or used methods developed by international or national groups who provide methods guidance (in published handbooks and or peer-reviewed articles) for developing and updating clinical guidelines, as well as prioritising clinical guidelines for updating.
Outcome
The publication reports any of the following data:
- ◦Description of update (or retirement) process (including roles and responsibilities at each stage)
- •types of update that exist or criteria used to determine if an update is necessary;
- •process for retiring a guideline;
- •criteria to prioritise which guideline is updated first;
- •criteria to prioritise which clinical questions within a guideline are updated;
- •evidence synthesis methodologies used to update clinical questions;
- •differences between review process for updated guideline versus original guideline;
- •differences between approval and endorsement process for updated guideline versus original guideline;
- •dissemination of updated guideline;
- •resources required to undertake update;
- •processes for living guidelines;
- •Evaluation of the process;
- •usability and or critique of the updating methodology;
- •timeliness, that is, specific processes that enable a more efficient and timely update.
Due to issues relating to transferability of guidelines developed for specific diseases, disease-specific publications (handbooks and or peer-reviewed publications which described, or had implemented, guidance for updating disease-specific guidelines) were excluded. Methodological handbooks were identified through searching the websites of eligible organisations and through screening the methodological handbooks included in Vernooij et al (2014). This was done by one reviewer (KC), and relevant handbooks identified were reviewed by a second reviewer (JQ) to confirm their eligibility.
All citations identified from the database search and through screening the peer-reviewed articles included in Martínez García et al (2017) were exported to Endnote, de-duplicated and then exported to Covidence (www.covidence.org) for screening. Two reviewers (KC, BC) independently screened titles and abstracts and full-texts applying the defined inclusion and exclusion criteria. Where disagreements occurred, discussions were held to reach consensus and where necessary, a third reviewer was involved (JQ). Citations excluded during the full-text review stage were documented alongside the reasoning for their exclusion (Supplementary file 3) and included in the PRISMA flow diagram.
Data extraction
Data were extracted from methodological handbooks and peer-reviewed articles by one reviewer (KC) and checked for accuracy and omissions by a second (JQ for methodological handbooks and BC for peer-reviewed articles). Where disagreements occurred, discussions were held to reach consensus and where necessary, a third reviewer was involved (BC for methodological handbooks and JQ for peer-reviewed articles). Data extraction was conducted in Microsoft Word, using predefined data extraction forms. The data extraction form was piloted first by one reviewer (KC) and checked by a second (JQ); it was then refined to include a section on living guidelines. Data extracted are listed in the outcome section of the eligibility criteria above.
Quality assessment
Methodological handbooks were quality assessed independently by two reviewers (KC and JQ) and any disagreements were resolved by deliberation, or if necessary, a third reviewer (BC). In the absence of an appropriate quality assessment tool specific to methodological handbooks or guidance, quality was assessed using the GIN-McMaster Guideline Development Checklist, which is a checklist of items to consider during the development of guidelines. Specifically, the six criteria relating to updating guidelines were utilised (GIN-McMaster, 2014).
Methodological quality of peer-reviewed articles was independently assessed by two reviewers (KC and BC) and any disagreements were resolved by deliberation, or if necessary, a third reviewer (JQ). This was completed using a modified version of the Appraisal tool for Cross-Sectional studies (AXIS) (Downes et al, 2016). Any disagreements were resolved by deliberation or, if necessary, a third reviewer.
Data synthesis
As the main data extracted for this review was descriptive in nature a narrative synthesis was undertaken.
Findings
Included handbooks and peer-reviewed articles
The search of 17 organisations’ websites (Supplementary file 1), together with the grey literature search resulted in the inclusion of 16 handbooks. The search of electronic databases (Supplementary file 2), from 1 January 2011 to 27 October 2021, identified 1,468 citations. A total of 36 full-text articles were assessed for eligibility and 33 were excluded (Supplementary file 3). This resulted in three articles being eligible for inclusion; no additional articles were identified from the index documents (see Figure 1).
PRISMA flow diagram of included studies
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
PRISMA flow diagram of included studies
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
PRISMA flow diagram of included studies
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Characteristics of included handbooks and peer-reviewed articles
A total of 16 handbooks from 11 organisations were included (Table 1 and Supplementary file 4). Four (SIGN, 2019; NICE, 2020a; 2020b; SIGN, 2021) were developed by organisations in the UK; three (IOM, 2011; Qaseem et al, 2019; USPSTF, 2021) by organisations in the US; five (Qaseem et al, 2012; Schünemann et al, 2014; WHO, 2014; Morgan et al, 2018; Sanabria et al, 2020) by international organisations and/or collaborations; and one each was developed in Estonia (EHIF, 2020); Germany (AWMF, 2013); Ireland (NCEC, 2019); and Switzerland (SCIH, 2011). One handbook (Sanabria, 2020), The UpPriority Tool, was included as an Appendix of one of the included peer-reviewed articles; this handbook described a prioritisation tool for updating clinical questions within a guideline. The remaining 15 handbooks described the process of developing de novo CGs, and included varying levels of detail on the updating processes used. Of the included handbooks, those produced by the Association of the Scientific Medical Societies in Germany (AWMF, 2013); National Clinical Effectiveness Committee (NCEC, 2019); American College of Physicians (Qaseem et al, 2019); Estonian Health Insurance Fund (EHIF, 2020); and US Preventative Service Task Force (USPSTF, 2021); provided details on most of the criteria outlined in the predefined data extraction table.
Summary of the data extracted from included handbooks
| Update types | Update triggers | Retire CG | CG prioritisation | CQ prioritisation | Evidence synthesis methods | Review | Approval | Disseminate | Resources | Living guidelines | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| IOM, 2011 USA |
N/R | ✔ | ✔ | N/R | N/R | N/R* | N/R | N/R | N/R | N/R | N/R |
| Swiss Centre for International Health, 2011
Switzerland |
Minor
Major |
✔ | N/R | N/R | N/R | N/R | N/R | N/R | N/R | N/R | N/R |
| GIN, 2012; Qaseem et al, 2012
International |
N/R | ✔ | N/R | N/R | N/R | N/R | N/R | N/R | N/R | N/R | N/R |
| AWMF, 2013 Germany |
Complete
Modular Key questions |
✔ | ✔ | N/R | N/R | ✔ | ✔◊ | ✔◊ | ✔◊ | ✔◊ | ✔ |
| WHO, 2014 | N/R | ✔ | N/R | ✔ | N/R | ✔ | ✔ | N/R | N/R | N/R | N/R |
| Schünemann et al, 2014
International |
Full
Partial |
✔ | N/R | N/R | N/R | N/R | N/R | N/R | N/R | ✔ | N/R |
| Development of rapid guidelines: Morgan et al, 2018
International |
N/R | ✔ | N/R | N/R | N/R | N/R | N/R | N/R | ✔ | N/R | N/R |
| NCEC, 2019 | Full
Rapid Partial |
✔ | ✔ | ✔ | N/R | ✔ | ✔ | ✔ | ✔ | ✔ | N/R |
| Sanabria et al, 2020
International |
N/A | ✔ | N/A | N/A | ✔ | N/R | N/A | N/A | N/A | ✔ | N/R |
| Qaseem et al, 2019
US |
Full Partial |
✔ | ✔ | ✔◊ | N/R | ✔◊ | ✔◊ | ✔◊ | ✔◊ | ✔◊ | N/R |
| SIGN, 2019
Scotland |
Update
Minor revision |
✔ | ✔ | N/R | N/R | ✔ | ✔ | N/R | N/R | N/R | ✔ |
| NICE, 2020b
UK |
Full
Partial |
✔ | ✔ | N/R | N/R | ✔ | ✔ | N/R | ✔ | N/R | N/R |
| NICE, 2020a
UK |
Rapid | ✔ | ✔ | N/R | N/R | ✔ | ✔ | ✔ | ✔ | ✔ | N/R |
| Estonian Health Insurance Fund, 2020
Estonia |
Full
Partial Individual questions |
✔ | ✔ | ✔ | N/R | ✔ | ✔◊ | ✔◊ | ✔◊ | ✔◊ | N/R |
| SIGN, 2021
Scotland |
Rapid | ✔ | ✔ | N/R | N/R | N/R | N/R | N/R | N/R | N/R | N/R |
| USPSTF, 2021
US |
Full
Targeted Reaffirm |
✔ | ✔ | ✔ | N/R | ✔ | ✔◊ | ✔◊ | ✔◊ | ✔◊ | N/R |
Key: ACP - American College of Physicians; AWMF - Association of the Scientific Medical Societies; CG - clinical guideline; CQ - clinical question; GIN - Guidelines International Network; IOM - Institute of Medicine; N/A - not applicable; NCEC - National Clinical Effectiveness Committee; NICE - National Institute for Health and Care Excellence; N/R - not reported; SIGN - Scottish Intercollegiate Guidelines Network; UK - United Kingdom; US - United States of America; USPSTF - US Preventative Services Task Force; WHO - World Health Organization. ◊Details not reported in handbook, but provided via email communication from the organisation.
Of the included handbooks, 13 specified that a review-by date indicated the need to update a guideline (range 3–5 years post-publication). Other update triggers identified included publication of new or contradictory evidence or guidance, expert opinion or feedback from those implementing the guideline, and changes in policy or legislation. The criteria used to prioritise which guideline to update first were reported in five handbooks; these included the rate of change of the evidence base on the topic, public health importance of the topic and cost of the condition. However, descriptions of the application of these criteria were lacking, as were the resources required.
Of the three peer-reviewed articles eligible for inclusion, one (published in 2020 (Casey et al, 2020)) was an evaluation of additional search techniques employed by NICE. While no associated handbook was identified for this evaluation, the authors state that these additional search techniques are applied routinely by NICE in guideline surveillance when required; this is implied by the methods described in Developing NICE guidelines: the manual (PMG20) (NICE, 2020a). The other two peer-reviewed articles (published in 2020 (Sanabria et al, 2020) and 2021 (Sanabria et al, 2021)) were evaluations of the UpPriority Tool developed by the GIN (Table 2 and Supplementary file 5).
Summary of the data extracted from peer-reviewed articles
| Publication description | Evaluation (as reported by authors) |
|---|---|
| Authors (year): Casey (2020)
Organisation: NICE Country: UK Search techniques: focused subject headings, subheadings, frequency operators and title only searches Databases searched: MEDLINE, Embase and PsycINFO Outcome measures: •Total number retrieved •Number of includes found •Search precision •Number Needed to Read |
Outcomes:
Round 1 testing Focused subject headings • Retrieval: MEDLINE, 1/69 not retrieved, Embase 3/78 not retrieved. • NNR: MEDLINE mean reduction = 11+/−14, Embase mean reduction = 85+/−74 Frequency operators • Retrieval: MEDLINE, 4 not retrieved, Embase 3 not retrieved. • NNR: MEDLINE mean reduction = 29+/−25, Embase mean reduction = 32+/−28 MeSH, Emtree, PsychINFO subheadings • Retrieval: MEDLINE, 3 not retrieved. • NNR: MEDLINE mean reduction = 8+/−14, Embase mean reduction = 71 +/−72, PsychINFO reduction for CG155 (from 53 to 29) and for CG142 (NNR from 30 to 27). Title only and the combination of title only and focused subject headings techniques • Retrieval: MEDLINE, 18 not retrieved, Embase 28 not retrieved. Round 2 testing: Combined searches using focused subject headings and frequency operators compared with baseline •Precision: 0.5% for baseline, 1.08% for combined (CG141) • NNR: 199 for baseline, 93 for combined (CG141) • Precision: 1.97% for baseline, 2.17% for combined (CG142) • NNR: 51 for baseline, 46 for combined (CG142) • Precision: 1.20% for baseline, 2.73% for combined (CG149) • NNR: 84 for baseline, 37 for combined (CG149) • Precision: 0.68% for baseline, 0.97% for combined (CG155) • NNR: 148 for baseline, 103 for combined (CG155) • Precision: 1.57% for baseline, 3.33% for combined (CG160) • NNR: 64 for baseline, 30 for combined (CG160) |
| Authors (year): Sanabria (2020)
Organisation: GIN Country: International Objective: To develop a pragmatic tool to prioritise clinical guideline questions for updating, the UpPriority Tool and pilot the tool with the NICE clinical guideline (Meningitis (bacterial) and meningococcal septicaemia in under 16s: recognition, diagnosis, and management). |
Usability/critique:
• 3 of 6 invited participants independently applied the tool. • Median 0.5 hours (range 0.5–2.0) to complete assessments using the tool. The overall ICC was 0.65 (95% CI 0.36–0.82). • The overall degree of agreement was considered fair, given the wide confidence intervals observed. • No changes were made to the tool at this stage. |
| Authors (year): Sanabria (2021)
Organisation: GIN Country: International Design: 30 appraisers systematically assessed 107 clinical questions from 4 guidelines developed in the Spanish National Health System clinical guideline program. Objective: to 1) use the UpPriority Tool to identify which clinical questions within the clinical guidelines need to be prioritised for updating and 2) assess the implementation of the tool in a real-world set of clinical guidelines. |
Usability/critique:
• The mean time each participant spent evaluating the all clinical questions with the tool was 3.8 hours (range 0.5 to 10 hours). • The degree of agreement among the participants was good for the clinical guideline on open-angle glaucoma (ICC 0.87; 95% CI 0.80–0.92), moderate for the clinical guidelines on chronic heart failure and inherited retinal dystrophies (ICC 0.62; 95% CI 0.80–0.92 and ICC 0.63; 95% CI 0.41–0.78, respectively), and poor for the clinical guideline on menopause (ICC 0.15; 95% CI -0.63 to 0.62). • As reported by the authors, after successfully applying the tool and considering the appraisers’ feedback, no changes in the tool were proposed. However, some areas for consideration when using the tool, included: 1. identification of key appraisers, 2. customisation of training materials, 3. establishment of priority thresholds, 4. provision of methodological support. |
Key: CI - confidence intervals; GIN - Guidelines International Network; ICC - intra-class correlation coefficient; NICE - National Institute for Health and Care Excellence; NNR - numbers needed to read; N/R - not reported.
Methodological quality of handbooks and peer-reviewed articles
In general, reporting of policies, procedures or timelines for routinely monitoring and reviewing whether the guideline needed to be updated was good across the included handbooks. Arrangements for guideline group membership and participation after completion of the guideline, as well as plans for funding and logistics for updating the guideline in the future, were poorly reported across the included handbooks (Figure 2 and Supplementary file 6).
Quality assessment of included handbooks
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Quality assessment of included handbooks
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Quality assessment of included handbooks
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
All three included peer-reviewed publications were of good quality. All provided a clear statement of aims, and methods were sufficiently described and study limitations presented. All studies employed an appropriate study design with appropriate outcomes and conclusions, which were justified by the results. All studies used convenience samples of guidelines that were representative of the target population, that is, guidelines requiring updating. No conflicts of interest were identified and all studies had ethical approval. Concerning the possible introduction of bias domains, the selection of guidelines and appraisers was representative of the target population and outcomes were measured appropriately. Some criteria in the AXIS tool were considered not applicable to the included studies (Figure 3 and Supplementary file 7).
Methodological quality assessment of included studies using modified AXIS tool
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Methodological quality assessment of included studies using modified AXIS tool
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Methodological quality assessment of included studies using modified AXIS tool
Citation: Evidence & Policy 19, 4; 10.1332/174426421X16854447463061
Discussion
Summary of findings
We identified 16 methodological handbooks (IOM, 2011; SCIH, 2011; Qaseem et al, 2012; AWMF, 2013; Schünemann et al, 2014; WHO, 2014; Morgan et al, 2018; SIGN, 2019; Qaseem et al, 2019; EHIF, 2020; Sanabria et al, 2020; NICE, 2020a; NICE, 2020b; SIGN, 2021; USPSTF, 2021) from 11 organisations that provided some guidance on the updating process of CGs, and three peer-reviewed articles that evaluated updating processes. Approaches to prioritisation between CGs, identified by this review, were broadly based on the topic and rate of change of the evidence base, the review-by date of the guideline, clinical burden and public health importance of the topic, and opinions of interested parties. The latter could potentially bias the outcome of the prioritisation process. The UpPriority Tool (Sanabria et al, 2020) was developed to prioritise the clinical questions (within a CG) that require updating; in turn, this may reduce the burden of updating the CG. The level of agreement for application of the tool across four CGs ranged from good to poor; which may reflect variation in the guideline development groups who originally developed the CGs (Sanabria et al, 2021). While feedback from the appraisers in the studies was positive and no changes to the tool itself were suggested, areas to consider for improvement included the identification of key appraisers, customisation of training materials, establishment of priority thresholds and provision of methodological support (Sanabria et al, 2021).
Like the index documents (Vernooij et al, 2014; Martinez Garcia et al, 2017) identified as part of this review, our findings showed that there is a need for detailed guidance for updating CGs, and methods and required resources to prioritise updating of clinical guidelines were variable and poorly reported. Furthermore, terminology across the handbooks included in this review was inconsistent, (for example, ‘complete update’ is used interchangeably with ‘full update’, as is ‘partial update’ with ‘modular update’). This lack of standard terminology and definitions is one of the challenges identified in updating CGs. To address this, the GIN Updating Guidelines Working Group compiled the Updating Glossary with domains, terms, definitions, and synonyms related to updating of CGs (Martinez Garcia et al, 2018). The authors concluded that use of the updating Glossary could facilitate and improve knowledge translation and enable identification of research gaps (Martinez Garcia et al, 2018).
The recommendations contained within a CG are typically underpinned by a systematic review (or evidence synthesis). In general, the evidence synthesis methodologies used to update CGs, as described in the included handbooks, were the same as those used to develop new guidelines. However, it is acknowledged that there is a need to improve the timeliness and reduce the burden of maintaining the validity of CGs. In a qualitative study of systematic review production models currently employed within and outside Cochrane (Turner, et al, 2017), six opportunities to improve the production of systematic reviews were identified. These included clarification of roles, active coordination of the project and improved capacity building and information sharing.
Updating an existing systematic review to inform an update to a CG is generally more efficient than starting a new systematic review. However, poor reporting of existing systematic reviews makes it difficult to update these reviews. Moreover, there is a lack of clear guidance on how updates to systematic reviews should be reported (Pieper D, 2017). In some instances, searches for an update to a CG differ from the original searches (Garner et al, 2016), which raises the issue of whether the updated searches should be applied to original dates as opposed to just the date from when the last search was conducted (Garner et al, 2016). This makes the process of updating CGs very onerous and there is a need to develop and evaluate more efficient search strategies (Martínez García L, 2012). In one included study (Casey et al, 2020), the authors investigated the impact of additional search techniques, employed by NICE, to determine if they increase precision and reduce screening burden without impacting on surveillance decisions. It was concluded that the search techniques employed should be considered for surveillance topics where the initial search yields a large number of studies for screening, and for rapid reviews where limited resources prohibit a full systematic review (Casey et al, 2020). Additionally, a previous study (Martinez Garcia et al, 2015) evaluated the efficiency and feasibility of two search approaches to determine if the recommendations within a CG require updating. The authors concluded that use of restrictive search strategies was a feasible and efficient method through which to identify significant new evidence likely to trigger a recommendation update.
Another issue associated with updating CGs is the lack of standardised reporting. The Checklist for the Reporting of Updated Guidelines (CheckUp) (Vernooij et al, 2017a) is a 16-item reporting guideline to evaluate the completeness of reporting in updated guidelines and to help guideline developers in updating CGs. The authors of CheckUp suggest assessing the quality of CGs using the AGREE II instrument and prioritising the update of high-quality CGs. The updating process could also incorporate improving the methodological quality of the CG (Vernooij et al, 2017b). A systematic assessment of the reporting of the updating process in updated CGs using the CheckUp tool found the reporting of the updating process in updated CGs to be suboptimal (Vernooij et al, 2017b). Another study found the presentation formats used to indicate the changes in recommendations varied widely across CGs, even within the same guideline organisation (Martínez García L, 2014).
While a number of COVID-19 living guidelines were identified in this review, these, and other disease-specific living guidelines, were not eligible for inclusion. Moreover, due to the need for rapid guidance in response to the COVID-19 pandemic, the methods used in these living guidelines were not included in organisations’ general, non-disease-specific methods guidance. Methods to develop and implement dynamic or living CGs are still in their infancy but they have been especially useful throughout the COVID-19 pandemic; for example ‘A living WHO guideline on drugs for COVID-19’, drawing on evidence synthesised in two living network meta-analyses (Rochwerg et al, 2020). For this review, only two eligible handbooks (AWMF, 2013; SIGN, 2019) described their universal approach to living guidelines. The National Health and Medical Research Council and NICE are in the process of developing their guidance for living guidelines. While it is anticipated that the use of living guidelines is becoming more common, guidance on routine use of living guidelines is lacking.
Strengths and limitations
We conducted an extensive grey literature search in conjunction with searches of the two main biomedical databases Medline and Embase. In addition, we contacted all identified organisations to retrieve non-published handbooks and or address gaps in the data extracted; therefore we believe that we included most of the existing relevant handbooks. However, there are some limitations. It is possible that we did not identify all relevant handbooks because some are not publicly available; and some may been published after the systematic search was conducted on 27 October 2021. Moreover, due to resource constraints, we restricted inclusion to English language only and did not search for, or include, disease-specific handbooks. It is possible that we did not identify all potentially eligible peer-reviewed articles from the database search. Finally, quality was assessed using the GIN-McMaster Guideline Development Checklist; this checklist has not been validated for the quality assessment of methodological handbooks, but was used in the absence of an appropriate tool.
Implications for practice
Of the handbooks included in this review, the following recommendations were noted:
- •Terminology and definitions used internationally should be standardised;
- •detail on the resources (time, funding, personnel) required to undertake an update to a CG, and who is responsible for each stage of the updating process, should be described;
- •methods used to determine if an update is indicated, as well as methods to prioritise CGs (and to prioritise clinical questions within a CG) for updating should be reported and standardised;
- •there is a need for differentiation between evidence synthesis methods used to update CGs and those used to develop guidelines de novo as using the same methods for both may represent inefficient use of resources.
Conclusion
This review identified 16 eligible handbooks from 11 organisations that described update processes and prioritisation methods for CGs; no single handbook contained guidance on all of our predefined steps. Those who provide methods guidance for developing and updating CGs should consider providing more comprehensive guidance and standardising the terminology used to facilitate optimal updating of CGs and prioritisation of CGs for updating.
Funding
This research was funded in part by the Health Research Board (HRB) under grant no. HRB-CICER-2016-1871. BC is funded by HRB Emerging Investigator Award [EIA-2019-09]
Acknowledgements
The authors would like to acknowledge the support of the Health Technology Assessment directorate at HIQA.
Authors’ contributions
KC: conceptualisation, methodology, writing – original draft preparation; JQ: conceptualisation, methodology, supervision, writing – original draft preparation; writing – review and editing; BC: conceptualisation, methodology, writing – original draft preparation; writing – review and editing; BT: methodology, writing – original draft preparation; writing – review and editing; MC: methodology, writing – review and editing; SS: writing – review and editing; MR: writing – review and editing; MON: conceptualisation, methodology, supervision, writing – review and editing.
PROSPERO registration number
CRD42021274400
Data availability statement
Data extraction tables are available as supplementary files, which are available on figshare (DOI: 10.6084/m9.figshare.20497167).
Supplementary files
Search strategies, data extraction tables and quality appraisal scoring are available on figshare (DOI: 10.6084/m9.figshare.20497167).
Ethics approval
Ethics approval was not required for this study as it involved a review of the published literature. No patients were involved in this study.
Conflict of interest
The authors declare that there is no conflict of interest.
References
AWMF Association of the Scientific Medical Societies (2013) Guidance manual and rules for guideline development, https://www.awmf.org/fileadmin/user_upload/Leitlinien/AWMF-Regelwerk/AWMF-Guidance_2013.pdf.
Cardwell, K., Quigley, J., Clyne, B., Tyner, B., Carrigan, M., Smith, S., Ryan, M. and O’Neill, M. (2021) Processes for updating guidelines: protocol for a systematic review, HRB Open Research, 4: 116, doi: 10.12688/hrbopenres.13448.1.
Casey, M., Raynor, M., Jacob, C., Sharp, S., McFarlane, E. (2020) Improving the precision of search strategies for guideline surveillance, Research Synthesis Methods, 11(6): 903–12, doi: 10.1002/jrsm.1461.
Clyne, B. et al. (2023) Perspectives on the production, and use, of rapid evidence in decision making during the COVID-19 pandemic: a qualitative study, BMJ Evidence-Based Medicine, 28(1): 48–57, doi: 10.1136/bmjebm-2021-111905.
Denyer, D., Tranfield, D. and van Aken, J.E. (2008) Developing design propositions through research synthesis, Organization Studies, 29(3): 393–413, doi: 10.1177/0170840607088020.
Department of Health (2019) How to develop a national clinical guideline: a manual for guideline developers, https://www.gov.ie/pdf/?file=https://assets.gov.ie/11532/e2424b86508c4b928b04cf2770fab528.pdf#page=1.
Downes, M.J., Brennan, M.L., Williams, H.C. and Dean, R.S. (2016) Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS), BMJ Open, 6(12): e011458, doi: 10.1136/bmjopen-2016-011458.
Estonian Health Insurance Fund (2020) Estonian Handbook for Guidelines Development 2020, Estonian Health Insurance Fund, https://ravijuhend.ee/uploads/userifles/Estonian_Handbook_for_Guidelines_Development_2020_copy.pdf.
Garner, P. et al. (2016) When and how to update systematic reviews: consensus and checklist, BMJ, 354: i3507, doi: 10.1136/bmj.i3507.
IOM Institute of Medicine (2011) Clinical Practice Guidelines We Can Trust, Washington, DC: National Academies Press, doi: 10.17226/13058.
Martínez Garcia, L. et al. (2015) Efficiency of pragmatic search strategies to update clinical guidelines recommendations, BMC Medical Research Methodology, 15: 57, doi: 10.1186/s12874-015-0058-2.
Martínez García, L. et al. (2017) Methodological systematic review identifies major limitations in prioritization processes for updating, Journal of Clinical Epidemiology, 86: 11–24, doi: 10.1016/j.jclinepi.2017.05.008.
Martínez García, L., Arévalo-Rodríguez, I., Solà, I., Haynes, R.B., Vandvik, P.O., Alonso-Coello, P. and Updating Guidelines Working Group (2012) Strategies for monitoring and updating clinical practice guidelines: a systematic review, Implementation Science, 7:109: doi: 10.1186/1748-5908-7-109.
Martínez García, L., Pardo-Hernandez, H., Sanabria, A.J., Alonso-Coello, P., Penman, K., McFarlane, E. and GIN Updating Guidelines Working Group (2018) Guideline on terminology and definitions of updating clinical guidelines: the updating glossary, Journal of Clinical Epidemiology, 95: 28–33, doi: 10.1016/j.jclinepi.2017.11.023.
Martínez García, L.M.E., McFarlane, E., Barnes, S., Sanabria, A.J., Alonso-Coello, P. and Alderson, P. (2014) Updated recommendations: an assessment of NICE clinical guidelines, Implementation Science, 9: 72, doi: 10.1186/1748-5908-9-72.
Morgan, R.L., Florez, I., Falavigna, M., Kowalski, S., Akl, E.A., Thayer, K.A. et al. (2018) Development of rapid guidelines: 3. GIN-McMaster guideline development checklist extension for rapid recommendations, Health Research Policy and Systems, 16: 63. doi: 10.1186/s12961-018-0330-0
NCEC (National Clinical Effectiveness Committee) (2019) How to develop a national clinical guideline: a manual for guideline developers, https://www.gov.ie/pdf/?file=https://assets.gov.ie/11532/e2424b86508c4b928b04cf2770fab528.pdf#page=1.
NICE National Institute for Health and Care Excellence (2020a) Appendix l, Interim process and methods for guidelines developed in response to health and social care emergencies, https://www.nice.org.uk/process/pmg20/resources/appendix-l-interim-process-and-methods-for-guidelines-developed-in-response-to-health-and-social-care-emergencies-pdf-11378590459333.
NICE National Institute for Health and Care Excellence (2020b) Developing NICE guidelines: the manual (PMG20), https://www.nice.org.uk/process/pmg20/chapter/introduction.
Page, M.J. et al. (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, BMJ, 372: n71, doi: 10.1136/bmj.n71.
Pardo-Hernandez, H. et al. (2020) UpPriority Working Group. The UpPriority tool: a prioritisation tool for the updating of clinical guideline’s questions, Abstracts of the 26th Cochrane Colloquium, Santiago, Chile, Cochrane Database of Systematic Reviews, (1 Suppl 1), doi: 10.1002/14651858.CD201901.
Pieper, D. and Mathes, T. (2017) Survey of instructions for authors on how to report an update of a systematic review: guidance is needed, BMJ Evidence-Based Medicine, 22(2): 45–8, doi: 10.1136/ebmed-2016-110609.
Qaseem, A., Forland, F., Macbeth, F., Ollenschläger, G., Phillips, S., van der Wees, P. and Board of Trustees of the Guidelines International Network (2012) Guidelines International Network: toward international standards for clinical practice guidelines, Annals of Internal Medicine, 156(7): 525–31, doi: 10.7326/0003-4819-156-7-201204030-00009.
Qaseem, A., Kansagara, D., Lin, J.S., Mustafa, R.A., Wilt, T.J., Clinical Guidelines Committee of the American College of P et al. (2019) Development of clinical guidelines and guidance statements by the clinical guidelines committee of the american college of physicians: update of methods, Annals of Internal Medicine, 170: 863–70. doi: 10.7326/M18-3290
Rochwerg, B. et al (2020) A living WHO guideline on drugs for covid-19, BMJ, 370: m3379, doi: 10.1136/bmj.m3379.
Sanabria, A.J., Pardo-Hernandez, H., Ballesteros, M., Canelo-Aybar, C., McFarlane, E., Nino de Guzman, E. et al. (2020) The UpPriority tool was developed to guide the prioritization of clinical guideline questions for updating, Journal of Clinical Epidemiology, 126: 80–92. doi: 10.1016/j.jclinepi.2020.06.018
Sanabria, A.J., Alonso-Coello, P., McFarlane, E., Niño de Guzman, E., Roqué, M., Martínez García, L. and UpPriority implementation working group (2021) The UpPriority tool supported prioritization processes for updating clinical guideline questions, Journal of Clinical Epidemiology, 139: 149–59, doi: 10.1016/j.jclinepi.2021.07.022.
Schünemann, H.J., Wiercioch, W., Etxeandia, I. et al. (2014) Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise, Canadian Medical Association Journal, 186(3): E123–42, 10.1503/cmaj.131237.
Scientific Medical Societies in Germany (2013) AWMF Guidance manual and rules for guideline development, https://www.awmf.org/fileadmin/user_upload/Leitlinien/AWMF-Regelwerk/AWMF-Guidance_2013.pdf.
Sharp, M.K., Tyner, B., Awang Baki, D.A.B., Farrell, C., Devane, D., Mahtani, K.R., Smith, S.M., O’Neill, M., Ryan, M. and Clyne, B. (2021) Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol, HRB Open Research, 4: 76, doi: 10.12688/hrbopenres.13325.2.
SIGN Scottish Intercollegiate Guidelines Network (2019) SIGN 50: a guideline developer’s handbook, https://www.sign.ac.uk/our-guidelines/sign-50-a-guideline-developers-handbook/.
SIGN Scottish Intercollegiate Guidelines Network (2021) Rapid Guideline Methodology, https://www.sign.ac.uk/media/1836/20210408-rapid-guideline-manual-10.pdf.
Swiss Centre for International Health (2011) Handbook for supporting the development of health system guidance, https://www.swisstph.ch/fileadmin/user_upload/WHOHSG_Handbook_v04.pdf.
Turner, T., Green, S., Tovey, D., McDonald, S., Soares-Weiser, K., Pestridge, C., Elliott, J., Project Transform Team and IKMD developers (2017) Producing Cochrane systematic reviews: a qualitative study of current approaches and opportunities for innovation and improvement, Systematic Reviews, 6(1): 147, doi: 10.1186/s13643-017-0542-3.
USPSTF US Preventive Services Task Force (2021) Procedure Manual, https://www.uspreventiveservicestaskforce.org/uspstf/about-uspstf/methods-and-processes/procedure-manual.
Vernooij, R.S.A., Sanabria, A.J., Solà, I., Alonso-Coello, P. and Martínez García, L. (2014) Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks, Implementation Science, 9: 3, doi: 10.1186/1748-5908-9-3.
Vernooij, R.W., Alonso-Coello, P., Brouwers, M., Martinez Garcia, L. and CheckUp Panel (2017a) Reporting items for updated clinical guidelines: checklist for the reporting of updated guidelines (CheckUp), PLoS Med, 14: e1002207, doi: 10.1371/journal.pmed.1002207.
Vernooij, R.W.M., Martinez Garcia, L., Florez, I.D., Hidalgo Armas, L., Poorthuis, M.H.F., Brouwers, M. and Alonso-Coello, P. (2017b) Updated clinical guidelines experience major reporting limitations, Implementation Science, 12: 120, doi: 10.1186/s13012-017-0651-3.
WHO World Health Organization (2014) WHO handbook for guideline development, 2nd edn, Geneva: World Health Organization, https://www.who.int/publications/i/item/9789241548960.
