Pressure for proof

Systematic reviews are emerging as the best test of gender treatment

Photo by Egor Myznik on Unsplash

New Zealand’s Ministry of Health has left open the possibility of a systematic review of the evidence for hormone suppression drugs used to interrupt the puberty of transgender-identifying children.

Last year the health ministry abandoned the claim on its website that puberty blocker drugs are “a safe and fully reversible medicine” to relieve the distress of children with the condition gender dysphoria. There is increasing international concern about the known and unknown risks of puberty blockers.

Following concerns raised by members of the public, NZ Ministry of Health officials last September suggested a three-step response: swift adoption of a more cautious website reference to puberty blockers; a medium-term rudimentary “evidence brief”; and a longer-term formal systematic review of the evidence base.

Discussion of a systematic review was put off, according to September 2022 email exchanges within the ministry obtained under the Official Information Act by Simon Tegg of the watchdog group Fully Informed.

Systematic reviews in Finland, Sweden, the United Kingdom and the American state of Florida have found the evidence base to be weak and uncertain, suggesting that puberty blockers are experimental, should no longer be offered as routine treatment for gender dysphoria and instead should be confined to formal research settings with ethical safeguards.

Systematic reviews are judged the most reliable way to assess the extent and quality of evidence for the effects of a medical intervention because they involve an exhaustive search of the relevant scientific literature using pre-determined rules to minimise the risk of bias. Without these features, an evidence review is open to cherry-picking of studies and results.

Asked for an update on New Zealand’s progress in adopting a more informed position on puberty blockers, a spokesperson for the health ministry said the evidence brief was expected to be released next month. (It had been expected this month.)

“The evidence brief is the first step towards reviewing the current evidence-base on safety and reversibility of puberty blockers,” the spokesperson told GCN last week.

“The ministry will decide in due course whether a systematic review needs to be commissioned.”

Graphic: Systematic reviews represent the highest quality of evidence

“Evidence-based medicine requires that clinical decisions should be based on the evidence in totality, and not on just a single study. Systematic reviews offer the best available evidence for decision-making in clinical practice. They are ‘the most reliable and comprehensive statement about what works’, and involve identifying, synthesising and assessing all available evidence by a systematic approach, to generate a robust, empirically derived answer to a focused research question”—Professor Sanjay Patole, University of Western Australia Medical School

Systematic silence

In Australia, as far as GCN is aware, there has never been a formal systematic review, run by an independent expert agency, of the evidence for puberty blockers, cross-sex hormone treatment or mastectomy for minors who reject their birth sex and identify as trans or non-binary.

GCN asked Australian health ministers if they would consider a systematic review of the evidence, and alerted their offices to the fact of overseas reviews.

The question was put by email to federal minister Mark Butler—whose government subsidises state children’s hospitals doing gender medicine—and his counterparts in NSW, Victoria, Queensland and Western Australia, those being the states with large, fast-growing gender clinic caseloads. Shadow health ministers were also asked.

No minister or shadow minister answered the question.

“One of the fundamental triggers for a change from [‘gender-affirmation’] to a more holistic [treatment] approach is an independent systematic review. That may be beyond the scope of a professional body [such as a medical society] and would probably be more likely sponsored by a government or university. Professional bodies, however, can’t ignore that there have already been four or five systematic reviews [of gender dysphoria care] internationally that have raised concerns about ‘affirmation’ and recommended holistic approaches. How many times do we have to re-create the wheel?”—an Australian medical academic, speaking to GCN on condition of anonymity

Psychiatry’s shift to caution

The Royal Australian and New Zealand College of Psychiatrists (RANZCP) did assess the state of the scientific literature as part of its 2019-21 review of its LGBTQ mental health policy. The methodology for the literature review has not been made public, but GCN understands it was not a systematic review.

It did result in a new, more cautious policy on gender dysphoria care that has aroused opposition from clinicians and researchers who subscribe to the dogmatic “gender-affirming” treatment approach, which they declare to be settled science and “life-saving”.

The RANZCP’s gender dysphoria policy, issued in August 2021, notes the “paucity of evidence” in this contested field of practice, and stresses the need for better data on outcomes, especially for children and adolescents.

The policy says “evidence and professional opinion is divided as to whether an affirmative approach should be taken in relation to treatment of transgender children or whether other approaches are more appropriate.”

In response, prominent gender-affirming figures from both countries last year published an article of protest in the Australian & New Zealand Journal of Psychiatry, with “queer non-binary researcher” Dr Sav Zwickl of the University of Melbourne as lead author.

The article claimed that “the RANZCP position runs directly counter to the well-established evidence base that gender affirmation improves health outcomes and strengthens quality of life.”

The authors urged health professionals to follow instead “evidence-based practice guidelines, such as those endorsed by [the gender-affirming lobbies] AusPATH and PATHA.”

AusPATH, the Australian Professional Association for Trans Health, endorses the 2018 “Australian Standards of Care” treatment guideline issued by the Royal Children’s Hospital Melbourne (RCH), which is home to Australia’s most influential youth gender clinic.

PATHA, New Zealand’s Professional Association for Transgender Health Aotearoa, stands behind the 2018 University of Waikato gender-affirming guideline.

Neither the NZ nor the Australian guideline makes reference to a systematic review; PATHA and RCH did not reply when asked to confirm there was no such review.

The RCH guideline did not grade the evidence invoked to support its treatment advice to clinicians, and claimed in a peer-reviewed position statement that, “The scarcity of high-quality published evidence on the topic prohibited the assessment of level (and quality) of evidence for these recommendations.”

That claim alone was enough to show the RCH document is “not a trustworthy guideline”, according to a pioneer of the evidence-based medicine movement, Professor Gordon Guyatt of Canada’s McMaster University.

One requirement for a trustworthy guideline is that its recommendations “are based on systematic review of the relevant evidence,” rather than a traditional narrative literature review in which “a bunch of experts write whatever they felt like, using no particular standards and no particular structure,” Professor Guyatt told the British Medical Journal.

“Australia needs a systematic review even though it’s likely to find what the others have [in Finland, Sweden, the UK and Florida]. There might be a little bit of new data every time [another systematic review] is done, but not enough to make significant changes, at least based on the articles I’ve seen. It is probably more of a trust issue, that the general public see the results as more trustworthy and applicable to them if it is done by [an institution] viewed as genuinely neutral”—an Australian clinical psychologist and researcher, speaking to GCN on condition of anonymity

Not all evidence is equal

The 2022 article by Dr Zwickl and gender-affirming colleagues objected that the RANZCP’s new dysphoria policy had failed to cite “important research” such as the 2017 Trans Pathways study, which is used by gender clinicians and activists to suggest that high rates of attempted suicide among trans-identifying youth justify early medical interventions.

The Trans Pathways study relied on an anonymous online survey and a “convenience”—or non-representative—sample. Its claim that 48 per cent of trans youth attempt suicide has been cited by the RCH foundation when appealing for donations to the hospital.

The RANZCP policy was defended earlier this year by University of Adelaide psychiatry academic Dr Patrick Clarke and independent researcher Dr Vanessa Spiller in a letter to the Australian & New Zealand Journal of Psychiatry.

They pointed out that not all studies are solid enough to be included in a literature review leading to a policy statement.

“Materials considered for inclusion in [policy statements] are evaluated according to the quality of their design and the methodologies employed, which provides a hierarchy of reliability through which findings can be interpreted,” they said.

They noted the Trans Pathways finding that 21 per cent of participants were unsure if they were intersex (a reference to rare medical conditions now known as disorders or differences of sexual development).

“This finding highlights reliability issues in anonymous, self-report, online community survey methodologies, with less than 1 per cent of people [in the general population] actually having a disorder of sexual development,” the Clarke-Spiller letter said.

“Similar methodologies fail to control for a variety of other factors such as involvement in therapy, concurrent use of psychotropic medications and so on, making it erroneous to unequivocally conclude that support and affirmation alone are the ‘protective factors against psychological distress, self-harm and suicidality’.”

Guides that misdirect

The cautious RANZCP policy was cited in February’s British Medical Journal investigation as an example of a health professional society turning away from early medicalisation of gender distress.

By contrast, the BMJ reported, gender-affirming medicalisation was presented in the United States as an uncontroversial standard of care endorsed by influential bodies such as the World Professional Association for Transgender Health (known as WPATH and affiliated with AusPATH and PATHA); the American Academy of Pediatrics (AAP) and the Endocrine Society.

WPATH’s latest guideline, issued last September, devoted a new chapter to adolescents, recognising the international surge in teenagers seeking medical interventions.

Yet the guideline authors protested there were still too few studies of this patient group to allow a systematic review of treatment outcomes.

Nor did the AAP’s affirmation-only policy statement, issued in 2018, involve a systematic review of the literature, the significance of which it misrepresented, according to a trenchant “fact-checking” article by clinical psychologist and sex researcher Dr James M Cantor.

The AAP has ignored paediatrician members who have questioned the rigour of its 2018 policy statement.

In 2022, a resolution moved by five members called on the AAP leadership to commission “a rigorous systematic review” on management of youth gender dysphoria.

Suggesting the AAP was out of step with an international shift to greater caution, they cited systematic reviews or new cautious policy statements in Finland, Sweden, England, France, Australia and New Zealand.

“Given the increasing numbers of children and young people identifying as transgender—as many as 9 per cent [in the US]—and the near unified movement away from hormonal and surgical interventions as first-line treatment in multiple countries, a review of the evolving evidence is imperative,” the resolution says.

The integrity of the Endocrine Society’s 2017 guideline has also come in for criticism from Professor Guyatt, as journalist Jennifer Block reported in her BMJ investigation of youth gender dysphoria—

“The Endocrine Society commissioned two systematic reviews for its clinical practice guideline, Endocrine Treatment of Gender-Dysphoric/Gender-Incongruent Persons: one on the effects of sex steroids on lipids and cardiovascular outcomes, the other on their effects on bone health. To indicate the quality of evidence underpinning its various guidelines, the Endocrine Society employed the GRADE system (grading of recommendations assessment, development, and evaluation) and judged the quality of evidence for all recommendations on adolescents as ‘low’ or ‘very low.’

“[Professor] Guyatt, who co-developed GRADE, found ‘serious problems’ with the Endocrine Society guidelines, noting that the systematic reviews didn’t look at the effect of the interventions on gender dysphoria itself, arguably ‘the most important outcome.’ He also noted that the Endocrine Society had at times paired strong recommendations—phrased as ‘we recommend’—with weak evidence. In the adolescent section, the weaker phrasing ‘we suggest’ is used for pubertal hormone suppression when children ‘first exhibit physical changes of puberty’; however, the stronger phrasing is used to ‘recommend’ [puberty blocker] treatment.

“ ‘GRADE discourages strong recommendations with low or very low quality evidence except under very specific circumstances,’ Guyatt told The BMJ. Those exceptions are ‘very few and far between,’ and when used in guidance, their rationale should be made explicit, Guyatt said. In an emailed response, the Endocrine Society referenced the GRADE system’s five exceptions, but did not specify which it was applying.”

Swedish restraint

In February last year, a few months after a systematic review, Sweden’s National Board of Health and Welfare urged “restraint” in the use of hormonal drug treatments for medicalised gender change, warning that the risks appear to outweigh the benefits.

In the recent past, the board had championed “gender-affirming” medical interventions as “safe and secure.”

Last month, the researchers involved in Sweden’s systematic review—from the Karolinska Institute, University of Gothenburg, Umeå University, and the Swedish agency for health technology assessment and assessment of social services—published a paper describing the results in the journal Acta Paediatrica.

They reported that the systematic review had found 9,934 potential studies on hormonal treatment of minors with gender dysphoria, but only 24 met the threshold for review, and these typically had methodological shortcomings.

Almost a dozen studies were excluded from the review as carrying too high a risk of bias, and among them was one of the two pioneering papers from the famous Amsterdam gender clinic. These papers, lacking control groups, were used to justify the international spread of paediatric medical transition.

In Acta Paediatrica, the Swedish researchers noted that there had been relevant studies published after the November 2021 cut-off date for their literature search, including one from the Amsterdam clinic tracking cross-sex hormone use among patients who began with puberty blockers.

Such studies could be candidates for inclusion in a future systematic review.

The researchers also pointed out the absence of randomised controlled trials—the gold standard for medical research—in gender dysphoria research.

“We call for such studies, which may be the only way to address biases that we have noted in the field,” they said.

Their review showed that the duration of treatment with puberty blockers or cross-sex hormones in the studies was rarely longer than four years,” the researchers said.

“The absence of long-term studies is worrying because many individuals start treatment as minors and cross-sex hormone treatment is lifelong.

“We could not evaluate the frequency of individuals who drop out from [puberty blocker] treatment and no longer wish to continue with gender transition.

“We know from internet-based surveys that detransitioning exists, but such studies cannot provide reliable estimates of detransitioning frequency because of selection bias.

“Studies that closely follow individuals who start [puberty blocker] therapy and/or cross-sex hormone treatment until at least age 30 are urgently needed.”

Note: GCN sought comment from the RANZCP, the PATHA executive and RCH