It’s time for transparency in Canadian clinical trial data

Science-based health professionals hold the scientific method is pretty high regard. We advocate for evaluations of treatments, and treatment decisions, based on the best research. We compile evidence based on fair tests that minimize the risks of bias. And we consider this evidence in the context of the plausibility of the treatment. The fact is, it’s actually not that hard to get a positive result in a trial, especially when it’s sloppily done or biased. And even when a trial is well done, there remains the risk of error simply due to chance alone. So to sort out true treatment effects, from fake effects, two key steps are helpful in reviewing the evidence.

1. Take prior probability into account when assessing data. While a detailed explanation of Bayes Theorem could take several posts, consider prior probability this way: Any test has flaws and limitations. Tests give probabilities based on the test method itself, not on what is being tested. Consequently, in order to evaluate the probability of “x” given a test result, we must incorporate the pre-test probability of “x”. Bayesian analysis uses any existing data, plus the data collected in the test, to give a prediction that factors in prior probabilities. It’s part of the reason why most published research findings are false.

2. Use systematic reviews to evaluate all the evidence. The best way to answer a specific clinical question is to collect all the potentially relevant information in a structured way, consider its quality, analyze it according to predetermined criteria, and then draw conclusions. A systematic review reduces the risk of cherry picking and author bias, compared to non-systematic data-collection or general literature reviews of evidence. A well-conducted systematic review will give us an answer based on the totality of evidence available, and is the best possible answer for a given question.

In order for our evaluation to factor in prior probability, and to be systematic, we need all the evidence. Unfortunately, that’s not always possible if evidence remains unpublished or is otherwise inaccessible. There is good evidence to show that negative studies are less likely to be published than positive studies. Sometimes called the “file drawer” effect, it’s not solely the fault of investigators, as journals seeking positive results may decline to publish negative studies. But unless these studies are found, systematic reviews are more likely to miss negative data, which means there’s the risk of bias in favor of an intervention. How bad is the problem? We really have no complete way to know, for any particular clinical question, just how much is missing or buried. This is a problem that has confounded researchers and authors of systematic reviews for decades.

Fortunately, it’s difficult to do a clinical trial without leaving any clues at all. You need to write a protocol, seek some sort of ethics board approval, find researchers to run your study, recruit patients, and finally, fund it. Assuming most investigators don’t set out to keep a clinical trial hidden from view, there may be other clues that become public knowledge. For example, at medical conference there may be hundreds of academic “posters” and dozens of rooms of researchers presenting research evidence – sometimes only the interim results of research studies. Only a fraction of this research ends up into the medical literature. Sometimes information emerges in lawsuits, or in drug approval applications to regulators like the FDA, who make this data available publicly. But searching for evidence shouldn’t rely on serendipity. To conduct the most accurate systematic review, we need a comprehensive system to track down every relevant clinical trial.

The simplest approach to finding relevant studies would be to register every single clinical trial conducted – ideally before the study begins, so that negative trials can’t subsequently be buried. Requiring a simplified registration at a central resource should not be too onerous a requirement for investigators planning to perform human experiments. Making this information publicly available at a centralized database would then allow researchers to determine how many trials had been performed, and could serve a starting point for a truly systematic review. It’s a simple and fairly elegant approach, so it should be no surprise that it’s been implemented already – albeit ineffectively. More that 25 years has passed since the first major call for a registry, yet even today we still don’t have an effective one. The pharmaceutical industry, led by GlaxoWellcome, started one voluntarily, but it was subsequently shuttered.  (GlaxoSmithKline has just recently announced plans to return to this level of transparency, to their credit.) The FDA passed legislation to require trial registration in 1997, and mandated the registration of “federally or privately funded clinical trials conducted under investigational new drug applications“. The well-known clinicaltrials.gov launched in 2000. In 2007, requirements were expanded to include all clinical trials (with the exception of early “phase 1” studies) for all products subject to the FDA’s regulation. There have been external pushes to require registration, including the International Committee of Medical Journal Editors who implemented this requirement in 2005 for any study considered for publishing. This looked like a good strategy – researchers that avoided registering would find themselves cut out from subsequently publishing in the world’s major medical journals. There was also the World Health Organization’s mandate to register in 2006, and then the World Medical Association’s Declaration of Helsinki, which establishes ethical principles for clinical trials, which was amended in 2008 to require that “Every clinical trial must be registered in a publicly accessible database before recruitment of the first subject.”

In light of this consensus throughout the research community that registries are a Good Thing, how effective have they been? Not as effective as you might think – only one-fifth of trials have actually been registered, based on an 2011 analysis. Don’t pin this all on the pharmaceutical industry. Among the different groups of investigators, industry-sponsored research was more likely to be registered than government- or other-funded research. It seems no-one is particularly good at walking the talk of registries, despite their obvious benefits to medical science. The idea is a good one, but the concept has failed from an implementation perspective. What’s needed is mandatory registration, multiple checks to find the non-compliant, and then enforcement of the requirement.

We know what we need, and we know how to do it. We have decades of international experience to look to. So this makes the recent Canadian Government announcement all the more puzzling:

The Honourable Leona Aglukkaq, Minister of Health, today announced the Government of Canada’s intention to create a web-based list of Health Canada authorized drug clinical trials in patients.
“This project will make information about drug clinical trials easier for Canadians to find and use so that they can make informed decisions about their health,” said Minister Aglukkaq. “It will no longer be necessary for patients and health care providers to search through various international registries to find out if trials are available in Canada.”

Stakeholders will be formally invited to provide feedback on the plan and proposed information to be included in the list later this fall. When ready, the list will be placed on Health Canada’s website. Health Canada will continue to encourage sponsors to register their clinical trials on the two publicly accessible registries currently available. These registries, which are also part of the World Health Organization’s Register Network, collect and display international clinical trial information and can be searched free of charge. Health Canada is committed to providing updates on progress related to clinical trial registration and disclosure, and feedback will be taken into consideration as the project moves forward. This initiative is another example of the Government of Canada’s commitment to transparency.

What’s the problem here? There are several, nicely itemized by Julia Belluz at the Science-ish blog:

1. There remains no mandatory registration requirement for all human clinical trials in Canada. While trials that are publicly funded or conducted at public hospitals must be registered at clinicaltrials.gov or another international registry, there is no requirement for other trials to be registered, including pharmaceutical industry-sponsored research. This requirement is in place in the United States.
2. There is little enforcement anticipated. As long as Health Canada’s approach will be limited to “encourage sponsors” it is difficult to expect that this will lead to full compliance. The FDA can levy investigators with fines of $10,000 per day – and yet even that penalty is not resulting in full compliance.
3. There is no intent to make the results of registered trials publicly available. Access to research information is what’s necessary to evaluate risks and benefits of therapies. It’s what GlaxoSmithKline has just promised. And this week, the British Medical Journal announced it will require this data disclosure – from all researchers as of January, 2013, for any trial published. Among the national regulators, Health Canada provides little transparency about the information it considers, or its rationale for its decisions. The FDA, in comparison, makes volumes of information available on every drug publicly available, including transcripts from meetings where drugs are evaluated for licensing.

What would true transparency, and a genuine commitment to improve patient care look like? Mandatory Canadian registration of all human trials, ideally on an international database, enforcement with real penalties, and reasonable public access to the the key research information used to guide regulatory decisions. Yes, as Matthew Herder pointed out in a response to Belluz’s original post, moving to this level transparency is not an easy matter. It’s a huge, organization-wide commitment, as the FDA’s initiative shows. But that’s the minimum standard we should be holding Health Canada to. While Health Canada has taken some tentative steps towards explaining the evidence it considers in making decisions about approved drugs, there is still a enormous chasm between Canada and international best practices. It compromises the evidence base, prevents the best science-based care, and ultimately, does a disservice to patients.

To get a sense of how out-of-step Health Canada is, one can look at the answers it provided to Julia Belluz in response to her original column.

What information will people see on the listing?
Say you were wondering what trials might be going on for prostate cancer, you can go to this list and see for that medical condition what trials Health Canada has authorized, what’s the drug, what’s the name of the company doing it. That would enable you to contact the sponsor, (to ask about) what trial is going on. . . It’s meant to provide information that would be useful for Canadians in making some informed choices about treatments around investigational drugs.

So, this doesn’t address the problem of the lack of oversight of trials?
Presently, in the regulatory framework, there’s no regulatory mandated requirement that clinical trial sponsors register their trial.

So there is no intent for the “web-based list” to be comprehensive, dramatically compromising its usefulness to researchers, health professionals, and the public.

Why is Canada so far behind other jurisdictions in addressing these issues?
If you look at the international community, there are registries that post the commencement of trials and sites. The disclosure of results, I don’t think it’s something that any regulatory body has completely resolved. We have heard those comments and are looking at that. I’m not sure it’s a component of this issue that any regulator has completely worked through. We stay in tune to the discussion on this, and we are watching what our international partners are doing. Our decisions will be informed by that.

[Note: America’s trials registry, clinicaltrials.gov, created a database for results reporting in 2008. This requires that trial results are put on the database within a year of completion of the trial.]

The conversations around creating a clinical trials registry in Canada have been happening since 2005. It’s 2012, and still no registry.
There has been dialogue, yes, and Health Canada has gone out to seek input from external stakeholders.

Now we have this listing, but still no registry.
It’s not meant to be a fulsome clinical trials registry. . . Health Canada is open to the idea of more information being made available, more transparency, certainly we’re looking at opportunities for people to give input, on how best to do that. Health Canada is embarking on a multi-year project to modernize the regulatory framework for food and drugs in Canada. The registration of clinical trials has been identified as an initiative that will be looked at under that process. There’s an ongoing process on how best to do that.

After seven years, Health Canada has nothing concrete to offer to Canadians with respect to transparent access and mandatory registration of trials data. It’s as if it is completely unaware of international best practices – of which it continues to fall further and further behind. In September, the European Medicines Agency, the EU equivalent to Health Canada, announced plans to proactively publish clinical trial data, and remarkably, allow access to full data sets. Even GlaxoSmithKline has just announced their support for the BMJ’s transparency requirements of making data available. To be clear, the entire industry isn’t on board yet, and while voluntary disclosure is laudable, unless disclosure rules are rigorous and enforced across all research, not just that sponsored by pharmaceutical companies, we’ll fail to capture all the trials.

Now enter the Canadian Senate. In a comprehensive report issued earlier this by the the Standing Senate Committee on Social Affairs, Science and Technology, entitled, Canada’s Clinical Trial Infrastructure: A Prescription for Improved Access to New Medicines, it made the following observation:

The committee agrees that drug development, as with all health research, must be carried out with the goal of generating an evidence-base upon which policy makers and health professionals can make the most informed decisions regarding the allocation of health resources. The transparency required in order to accomplish this will not be attained through ‘soft governance’, as Trudo Lemmens, Scholl Chair in Health Law and Policy at the University of Toronto, emphasize when he appeared before the committee. Transparency will only be achieved by making registration of clinical trials mandatory. (emphasis added)

And its advice is specific:

The committee therefore recommends that the Minister of Health:

• move to immediately require clinical trial registration to the greatest degree permitted under its existing legislative and regulatory authorities;
• determine and propose the necessary amendments to the Food and Drugs Act and/or the clinical trial regulations contained in Part C, Division 5 of the Food and Drug Regulations, to require that manufacturers register a comprehensive set of data for clinical trial phases II and III on a WHO recognized website prior to recruiting any participants. Registration must include, but not be limited to, all results, adverse reactions, withdrawal of participants (nonidentifying), and prematurely ended trials;
• require that all foreign clinical trials that are used to support applications for market authorizations in Canada have met equivalent registration standards; and,
• implement measures to strictly enforce this recommendation in order to ensure transparency of the clinical trial process and of the processes at Health Canada.

Conclusion

Health Canada’s recent announcement of a “web-based list of authorized clinical trials” is regrettably out of step with Canadian expectations and international best practices. We know the evidence base is skewed. We know how to improve it. The benefits of open data to the health of Canadians, and patients worldwide, are clear. Yet after more than a decade of discussion, Health Canada has nothing substantive to offer to Canadians who are seeking greater transparency in clinical trials data. It’s unfortunate, and it needs to change.

Photo from flickr user Roche photo (no, not that Roche) used under a CC licence.