Registry data for use in health technology assessments in Norway – Opportunities and challenges

Decisions on uptake of medicines and medical devices on health insurance schemes are increasingly based on a health technology assessment (HTA) process. In Norway, the process has included reimbursement of out-patient medicines for two decades. During the past years, in-patient medicines, medical devices and more recently vaccines are all included in the HTA system. In the present article, we outline the Norwegian HTA system including its central components and partners. HTA as a scientific approach puts evidence on efficacy, safety and value of interventions into a broader perspective, explicitly considering relevant factors, among these legal, ethical and organisational aspects of the intervention. Although several combinations of aspects are possible, the most common is an assessment of the relative effectiveness and cost-effectiveness of the intervention. Randomised controlled trials have long been considered the mainstay for assessment of clinical effectiveness pre-launch, while registry data have been used to inform safety post-lunch, by pharmacovigilance. Recently, we have seen a move towards more use of “real world evidence”, i.e. data from non-RCT sources, mainly from registries. A model-based approach is often used to assess cost-effectiveness, in this context, different types of evidence from different sources are often synthesized. In this paper we describe the central components of HTA with special emphasis on

This is an open access article distributed under the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

MEDICAL DEVICES
For a new medicine to be "approved" for use in Norway, it needs regulatory approval from the European Medicine Agency (EMA) and/or the Norwegian Medicine Agency (NoMA). These are comparable to the regulatory body in the United States, the U.S. Food and Drug Administration (FDA) (1). Most newer medicines go through a centralised procedure in EMA. Regulatory approval assesses whether the medicine is likely to generate more health benefits than harm, i.e. whether the medicine is likely to display a positive risk-benefit ratio.
Medicines are usually tested in several phases, labelled phase one, two, three and four trials. Phase one and two are often uncontrolled studies conducted in humans aimed at assessing the safety and tolerability of the new medicine and to assess different dosing regimens. Phase two studies sometimes contain a control group and can be designed to give an early indication of clinical effectiveness. Phase three trials are often randomised controlled studies, the pivotal trials evaluating clinical effectiveness and safety of the treatment. Phase four trials are conducted post launch, are usually not controlled, often registry based and often aims at collecting either more information of safety or to assess relative effectiveness against other comparators than used in the phase three trial. Data from phase one, two and three are assessed in the regulatory process leading up to a decision to grant or decline an application for marketing authorisation.
A regulatory system is also in place for medical devices, but the evidentiary requirements are generally less strict than for pharmaceuticals, where randomised controlled trials (RCTs) are usually required (2,3). Granting of a marketing authorisation (MA) means that the medicine or medical device can be prescribed and sold in the relevant jurisdictions. For a medicine to be marketed in Norway, a list price must also be present.
Norway has a public, tax financed healthcare system, where the state act as both an insurer and provider of care (4). For a drug to be adopted or "approved" for uptake on public financing, each intervention and each indication must go through a value assessment or a health technology assessment (HTA), in addition to the regulatory approval. In most instances, all or some of the data used for marketing authorisation are also used to assess relative effectiveness in the HTA, but seeing that the research question is different (i.e. a different comparator or a different outcome may be judged relevant), the conclusions may vary between the regulatory agency and the national HTA body. While the regulatory body asks whether the risk-benefit ratio is positive, the HTA assessment asks whether the new treatment is more effective than existing treatments and if so, if the added benefit is worth the added costs.
The International Network of Agencies for Health Technology Assessment (INAHTA), Health Technology Assessment international (HTAi) and other partner organisations have collaborated to create a glossary for terms related to health technology assessments. A health technology assessment is here defined as: "A multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle. The purpose is to inform decisionmaking in order to promote an equitable, efficient, and high-quality health system." (5). Although most applications of health technology assessments are performed for pharmaceuticals and medical devices, health technologies are in this context broadly defined as any drug, device, procedure, organisational aspects or other technology for treatment, diagnosing or care of patients (6). As seen in the HTA definition, the prosses aims to aid decision making.
In Norway, a set of priority setting criteria are used to guide decision on which technologies should be included in public financing schemes. Currently, the Norwegian priority setting criteria are formulated as disease severity, health benefit and costs (7). In terms of quantifiable units, disease severity and health benefit are both measured in quality adjusted life years (QALYs), while costs are measured in monetary terms (NOK). In practice, these criteria are assessed based on a health economic evaluation, more specifically a costeffectiveness analysis which reports incremental costs (NOK) per QALY gained. In order to see exactly how these priority setting criteria are informed by a HTA report, consider that although HTA is defined as a process, a HTA report is defined as "A report that includes a comprehensive systematic literature review, or a systematic review of high level evidence, evaluating the safety and effectiveness of a technology, as well as an analysis of the cost-effectiveness of the techno-logy…." (5).
Hence, it is the health economic evaluation part of the HTA report that directly informs the Norwegian priority setting criteria based on their current formulation. In situations where a health economic analysis is not present, the criteria cannot be formally quantified. For use in clinical settings or in settings where a national HTA report is not available, the criteria do however have a textual description (7).

The Norwegian system for payer decisions on health technologies
HTA based decisions regarding adoption of out-patient medicines on public financing was introduced in Norway at the beginning of 2001 through the "blue prescription pathway" ("blåreseptordningen") (8). In this system, NoMA assesses HTA dossiers submitted by the holders of marketing authorisations and acts as a decision maker as long as the budget implications of adoption does not exceed a predetermined limit (formerly "bagatellgrensen", now "fullmaktsgrensen"). At this point of time, there was no formal system for evaluation of in-patient medicines, medical devices or other technologies, hence these were adopted without prior evaluation.
Acknowledging that in-patient technologies may also be subject to priority setting dilemmas, a national council for priority setting in healthcare was established in 2007 (9). The council had no formal decision-making power, hence acted more as an advisory board. Some board members did however have decision making power in their own organisations and were supposed to make sure that recommendations were followed in their jurisdictions. As basis for many of the technologies discussed, HTA reports were often commissioned from the national HTA centre (in 2004, the former SMM was merged with other units and became the Norwegian Knowledge Centre for the Health Services (9), in 2015 this unit was moved into the Norwegian Institute of Public Health (NIPH)).
A formal national HTA and decision-making process for in-hospital technologies was established in 2013, with the creation of The National System for Managed Introduction of New Health Technologies within the Specialist Health Service (10,11). With the introduction of this system, an HTA process gradually became mandatory for all new indications for pharmaceuticals, medical devices and other in-hospital technologies. To date (March 2021), the system has mainly focused on introduction of new products at launch. Processes for reevaluation and potential disinvestment of included technologies are currently under development. A HTA process has recently been introduced for vaccines (12), including the establishment of a scientific reference group for the vaccination programs at NIPH. Other public health interventions and non-drug interventions offered in out-patient care are currently introduced without prior HTA evaluation. We will briefly describe the different components below. More general methods for systematic reviews and health economic evaluations have previously been described in this journal (13) and will not be described in detail here.

Structure of and stakeholder roles in the national HTA system
The system for in-patient technologies is a formal decision-making process based on horizon scanning, HTA evaluation, national priority setting decisions and implementation (10).
Horizon scanning: A horizon scanning system is set up in order to identify new technologies eligible for assessment at an early point of time. For pharmaceuticals and medical devices, the scanning is linked to the regulatory process that precedes HTA evaluation. For other technologies, early identification is more challenging.
HTA evaluation/HTA reports: The system has two separate processes for HTA evaluation. The default option is that the manufacturer submits an HTA dossier informing relative effectiveness and cost-effectiveness that is reviewed by NoMA in case of pharmaceuticals and NIPH in case of non-drug technologies. NoMA and NIPH then assess the documentation for clinical-and cost-effectiveness and the assessment is published in a single technology assessment report ("hurtigmetodevurdering"). These reports assess one new pharmaceutical or one new medical device. In this part of the system, the manufacturers have an important role, in the sense that they are responsible for development and submission of HTA dossiers. This system is inspired by and similar to the UK single technology assessment system (14).
The alterative and much less used option is that NIPH can be commissioned to do a multiple technology assessment (MTA) ("fullstendig metodevurdering"), which includes development of de-novo health economic model and analysis as well as an own systematic literature review, often including a direct or network meta-analysis. Other dimensions (ethical, legal or organisational aspects) may also be included. This is similar to the UK multiple technology assessment and is used in cases where either it is unfeasible for a manufacturer to develop a HTA dossier or in cases where comparisons across several treatment options is relevant.
As an example, NoMA considered clinical effectiveness and cost-effectiveness of each new treatment separately for relapsing-remitting multiple sclerosis based on the manufacturers submitted documentation (e.g. cladribine (15)), while NIPH later considered relative clinical effectiveness and cost-effectiveness and also legal and ethical aspects of several treatments based on analyses conducted in-house for the same indication some years later (16)(17)(18).
National priority setting decisions: A commissioning forum consisting of the directors of research at the four regional health authorities commission HTA reports based on horizon scanning or suggestions from relevant stakeholders. HTA reports are then developed by either NoMA or NIPH. Finished reports are cleared by the commissioning forum. A Decision Forum consisting of the Chief Executive Officers from the four regional health authorities (19), will make national priority setting decision on the basis of the HTA reports and other considerations. The decisions made by the Decision Forum act as an instruction to the hospitals on which methods to adapt. When appropriate, the system is also linked to procurement.
Implementation: for pharmaceuticals, the uptake is formally monitored.

Data sources for health technology assessment
A health technology assessment may contain a systematic assessment of the clinical effectiveness, safety, health economic, legal, ethical and organisational aspects of a method, c.f. note to full definition of HTA (5). Although a health technology assessment is defined to contain many possible aspects, only a selection of these dimensions is considered in each case, a HTA report usually contain a combination of clinical-and cost-effectiveness.
Most common research questions are in other words: 1. Is the new intervention clinically effective relative to current practice on relevant clinical outcomes for the relevant population? 2. Can the intervention be considered a cost-effective alternative to the comparator for the relevant population?

Clinical effectiveness and safety
Given the research questions outlined above, what data sources are most used and what are the alternatives? Traditionally, both regulatory and HTA agencies have relied mainly on pivotal randomised controlled trials to assess the efficacy of new treatments (20), while registry data have been used primarily for assessment or follow-up safety (20,21). This approach is however changing. There is a growing acceptance that although RCTs do provide the highest level of evidence for the causal effect of an intervention on an outcome, there may be several circumstances where these types of trials are impractical, too expensive or too difficult to conduct. Among these circumstances are difficulties finding enough patients (rare diseases), difficulty blinding (some medical devices and procedures) and ethical concerns. In the context of regulatory or payer approval of medicines or medical devices, supporting data from other sources than clinical trials, is often labelled real world data (RWD) or real-world evidence (RWE) (22,23). The term RWD is often used as an umbrella term covering many different data sources and potential usages (22), however data from registries seems to be the most common source for RWD (22,24).
Due to perceived need for new treatment options in some disease areas, regulatory agencies have reduced the evidence burden on submitting companies by accepting more uncertain evidence, i.e. smaller trials, trials without control arms, crossover trials and increased used of surrogate outcomes, "In limited instances, FDA has accepted RWE to support drug product approvals, primarily in the setting of oncology and rare diseases." (21).
This trend towards study designs that to a lesser degree can inform a causal relationship between intervention and outcome, have spilled over to the availability of evidence for health technology assessments and has generated an increased interest in both regulatory and HTA bodies for supporting evidence. Many of the current applications relates to the need to generate e.g. an artificial control arm for an uncontrolled study (25). The EMA also list among possible usages of RWD "Studies based on patient registries may also contextualise the results of uncontrolled trials, provide comparator groups of patients for a single arm trial on a case-bycase basis where a randomised controlled trial (RCT) is deemed not feasible or unethical, and support registrybased randomised controlled trials (RRCTs) for patient recruitment…" (26).
Some organisations distinguish between real-world data and real-world evidence, i.e. distinguish between the raw data and data that has been analysed and contextualised (21,24). The FDA defines the two concepts in the following manner: • "Real-World Data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. • Real-World Evidence (RWE) is the clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD." (21).
As RWD is increasingly seen as a necessity, many regulatory and HTA agencies have issued guidances for the collection, analysis and use of such data, among these are the FDA (21), the EMA (26), the Institute of Clinical and Economic Review (ICER) (27)

and the International Society for Pharmacoeconomics and Outcomes
Research (ISPOR) (20,28,29). Reviews of practices in different HTA organisations reveal that agencies differ with regard to use of RWD (30)(31)(32). A common feature is that data from RCTs are considered a higher level of evidence than RWD for questions of treatment effectiveness, but that RWD may be considered as either a supplement or in situations where RCTs are difficult. Similar is also the use of RWD for informing both natural history parameters (incidence, prevalence, etc.) and resource use or costs for use in cost-effectiveness analyses (32). Similarly, a review of health economic evaluations performed by ICER, reveals that RWE is used to inform natural history parameters (disease progression input) and resource use or costs and only on occasion to inform clinical effectiveness (33).

Cost-effectiveness analyses
While the systematic review informs questions about clinical effectiveness, the health economic report assesses whether the treatment can be considered to fulfil the priority setting criteria (Figure 1). A health technology may e.g. produce more benefits than harms (i.e. get a positive regulatory decision) while at the same time being considered less effective and more costly than existing treatment alternatives. A situation like this may arise if e.g. choice of outcome or comparator differ between the regulatory and HTA body. Similarly, the technology may produce more benefits than existing therapies, but the added benefit may be judged to be too small to justify the added costs (i.e. get a negative decision based on HTA).
A health economic evaluation can be based exclusively on secondary data, or exclusively on primary data or as a combination of the two. A health economic model will synthesize evidence from different sources and will be used in cases where the whole analysis cannot be based solely on primary data. Model based health economic evaluations are most common, both in HTA reports and in the published literature (34). A health economic model can be viewed as a simplified patient care pathway, that aim to capture all relevant effects of the intervention on resource use and health outcomes. If the intervention affects mortality, the model will often take a lifetime perspective.
To populate the health economic model, a range of different types of information is usually needed ( Figure  1). Broadly, one needs information on the natural history of disease (epidemiological data), clinical effectiveness of the intervention, quality of life weights for included health states or events, costs connected to included health states and events and cost of the inter- Stippled lines indicate possible data usage that is either not commonly used or not commonly recommend as a first choice. vention and comparator. These different classes of input data will have different preferred data sources and designs.
Use of registry data is preferred for information regarding natural history of disease (35). Norway has several mandatory national health registries covering the whole population through registration of all contacts with the health care system. These registries typically serve distinct purposes for the government, among these quality control and epidemic preparedness. The other dominant type of registry is quality registries, with special focus on specific diseases, such as for instance hip fracture or stroke. These quality registries often have more detailed information, providing in-depth knowledge about the disease or condition in question. Some of these registries gather data that may require consent from participants, possibly introducing selection bias.
If registry data are not available, information from RCTs may be substituted. As an example, a mortality rate or baseline rate of a clinical event (e.g. rate of myocardial infarction) would be preferred based on national registry data, because this would likely to be a better representation of the local context than a rate based on the control arm from a RCT. The reason for this is that the rate in the control arm would likely be from a highly selected population (due to strict inclusion criteria in the trial) and also likely be from a different country, with a different risk profile due to genetic, environmental factors or current treatment patterns. If a Norwegian registry-based rate of myocardial infarction was not available, one could use information from a registry in a jurisdiction where organisation of health services, available treatments and potentially other factors are as similar as possible. In this context, a rate from an RCT would be considered to be a "lower level of evidence" for this input due to the possible low external validity of the estimate (Figure 2). If no published data are available, local clinical experts may provide more representative estimates than a potentially unsourced estimate from a published economic evaluation concerning a different jurisdiction.
For chronic diseases, incidence rates are regularly published from Scandinavian countries. Rates or hazards of later events (e.g. risk increase of a second myocardial infarction following the first event or rate of heart failure following a myocardial infarction), may be harder to come by, more research on patient prognosis would strengthen this part of the HTAs.
In the context of HTA, clinical effectiveness data for a health economic evaluation will preferably be obtained from a systematic review of randomised controlled trials (35). Registry data or other observational data is however used in cases when RCT data are not available (35). Large representative registries or cohorts would generally be preferred from a health economic perspective. Estimates of relative effectiveness is generally regarded as more transferable between jurisdictions than absolute differences. Hence, creating models reflecting natural history of disease based on local data and thereafter multiplying by an estimate of relative effect from a trial is recommended (36).
In addition, registries may inform health utility weights, when available. To date, only a few registries or large population studies gather data on health-related quality of life, among these are the Norwegian hip  (35)).
The ranking in the pyramid is not definitive but can guide selection of studies. The aim is to find a study with a strong design (to minimise bias), while also considering the representativeness of the study population for the population under assessment.
fracture registry and the Tromsø study (37,38). These large representative data sets facilitate the use of local HRQoL estimates in HTA reports.
Registries are also a very good source for resource use and costs of health events and states. If feasible in the specific project, information on resource use and costs can be gathered specifically for the evaluation in question (39). As with natural history of disease parameters, raw data or real-world evidence (i.e. publications) from similar countries with similar organisation of health care services and treatment practices could also be used, an example from Sweden was for example used in a HTA report on stroke prevention (40).

Other applications
Although RWD is mainly used for assessment of clinical effectiveness, safety and cost-effectiveness, there are also other related areas of application relevant to HTA. As briefly mentioned above, registries can be used to gather information on resource use and costs for economic evaluations. One related area of application is in budget impact analyses or other analyses of the economic burden of disease, one example is a recently published article on the economic burden of herpes zoster in Norway (41), another is an analysis of the economic burden of HPV (42).
Registry data may also inform current treatment patterns, e.g. what would be the relevant comparator in the Norwegian setting, treatment switching (43), compliance with prescribed treatment and other aspects of the use of the technology or comparator in a "real life" setting.

DISCUSSION
Registries can be used for reassessments (44), i.e. to assess real-life effectiveness and safety. Considering that RCTs are the most unbiased estimate of the relative effect of an intervention, registry data can be used as a supplement in order to shed light on whether the efficacy estimates from trials hold up in "real life". Examples of these types of studies have recently been published for bisphosphonates for the prevention of osteoporotic fractures (45) and direct oral anticoagulation therapies for prevention of stroke (46). Most HTAs (i.e. single technology assessments, "hurtigmetodevurderinger") are performed at product launch, using mainly data from pivotal randomised trials. One could imagine reassessing technologies using registry data in multiple technology assessments ("fullstendige metodevurderinger") at later points in the product lifecycle, in order to capture the "real life" performance of the interventions. Although there are clearly opportunities, there are also challenges. Considering that oncology treatments often receive regulatory approval based on limited data, these would also be the treatments where the potential benefit of reassessment would be substantial. One limitation for oncology medicines is however that they are not currently part of the Norwegian Prescription Database (NorPD), limiting the feasibility of these analyses. Both the recent INSPIRE project (47) and the planned expansion of NorPD to include inhospital pharmaceuticals do however show great promise for the future. A related usage of registry data is pragmatic clinical trials, where information from registries is used for primary or secondary data collection. This type of analysis has been highlighted as one important future direction for both clinical trials and cost-effectiveness analyses (48).
Some have attempted to assess and extend the generalisability of RCT to local settings using observational data (49). Others have attempted to combine observational and experimental evidence in order to adjust RCT evidence to better fit local data, by estimating treatment effects using causal inference (50). Further development of these methods and incorporation of these into HTA could be an important future step. Another important trend is the recent rise of methods for causal inference in analysing observational and experimental data, this will likely be an important development for relative effectiveness research.
One possible future direction is to increase collaboration between epidemiologists and HTA researchers, in order to fully integrate information from registers and population studies into the evidence base used for decisions. Some examples exist (51,52), but there is clearly a potential for further interdisciplinary collaborations to bring forth decision relevant information.
Other factors that have proven to be important previously and which will also have impact in the future, includes technological developments and privacy issues. Following the introduction of GDPR in most of Europe, there has been a strong focus on issues that both make research more cumbersome and potential increased efficiency. Following the GDPR introduction in Norway, a new platform for storing and analysing health data will include numerous registries and sources of health data is under development (53). As defined by the Norwegian Ministry of Health and Care Services, the main goal of this platform will be better use of all the publicly funded data sources (54).
Use of machine learning and artificial intelligence is increasing in several research areas, including HTA (55,56). These techniques are also moving into the field of epidemiology and can be used for instance in searching for new, unidentified relationships in large data as reported from combining registries and other data sources. Although this in the beginning may be more as a form of identifying potential hypotheses that later can be further examined using standard methods, there are also developments in the direction of thinking causally in machine learning (57).

CONCLUSION
Data from registries are key in the development of robust HTA reports and the usage of registry data in this context is likely to increase in the years to come. We see great potential in interdisciplinary collaboration to further strengthen the evidence base for health policy decision on adoption and reassessment of health technologies.