Scientists Replicate Studies for the Reproducibility Project: Cancer Biology

Rigor and reproducibility are essential to the ability of scientific discoveries to advance scientific knowledge and to enable its translation into therapeutics and diagnostics in the clinic.  Senior staff researchers of the Center for Developmental Therapeutics (CDT), Irawati Kandela, Fraser Aird, and Christine Mantis, are playing a significant role inThe Reproducibility Project: Cancer Biology through their efforts to reproduce the findings of three seminal studies in the field of cancer biology.  These replication studies were the among the first five associated with the Reproducibility to be published on January 19th in eLife.  Commentaries on their findings were published in major scientific journals, including Science and Nature and earned a featured place in NPR’s Science Friday broadcast and an article in the Washington Post “ Researchers struggle to replicate 5 influential .

The studies replicated by the Center for Developmental Therapeutics were previously published as: “BET bromodomain inhibition as a therapeutic strategy to target c-Myc”, Cell 2011,146: 904-917;  “Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs”, Science 2010, 328:1031-1035; and “Discovery and preclinical validation of drug indications using compendia of public gene expression data”, Science Translational Medicine 2011, 3:96ra77.

The Reproducibility Project: Cancer Biology is a collaboration between Science Exchange and the Center for Open Science, and is independently replicating a subset of experimental results from a number of high-profile papers in the field of cancer biology studies published between 2010-2012 using the Science Exchange network of expert scientific labs.

CDT was selected to receive funding from the Reproducibility Project to replicate these studies.  They were required to first develop a Registered Report detailing the proposed experimental designs and protocols for the replications, which was then peer reviewed and published prior to replicating the original studies. Once the report was approved they embarked on the Replication Study experiments using the same protocols and materials as the original published study.  Once the replication studies were completed they were submitted for final review before publication.

Despite the use of exacting study design CDT scientists were unable to match the results of the original studies.  Their studies failed to demonstrate significant efficacy of the compounds the new therapeutics tested in the original studies for treatment of cancer in animal models.  Copies of the registered reports and the CDT replication studies can be found at https://elifesciences.org/collections/reproducibility-project-cancer-biology.

For the CDT scientists, this experience highlighted the importance of providing detailed information about the methods, protocols and reagents used when publishing scientific data.  They found that replication of the original studies was confounded by lack of detailed information and information gaps in the original publications that generated important questions requiring clarification from the original study authors, which were met with varying degrees of co-operation.

The Center for Developmental Therapeutics (CDT) was established to rapidly and efficiently advance novel therapeutic interventions from basic research to the clinic. The CDT is one of the first academic-based drug development centers that bridges all aspects of translational research in a single entity including early drug discovery, mechanistic research, pre-clinical development and translation into the clinic.

 

 

Abstracts:

Replication Study: BET bromodomain inhibition as a therapeutic strategy to target c-Myc 

https://elifesciences.org/content/6/e21253

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper "BET bromodomain inhibition as a therapeutic strategy to target c-Myc" (Delmore et al., 2011). Here we repor­­t the results of those experiments. We found that treatment of human multiple myeloma (MM) cells with the small-molecular inhibitor of BET bromodomains, (+)-JQ1, selectively downregulated MYC transcription, which is similar to what was reported in the original study (Figure 3B; Delmore et al., 2011). Efficacy of (+)-JQ1 was evaluated in an orthotopically xenografted model of MM. Overall survival was increased in (+)-JQ1 treated mice compared to vehicle control, similar to the original study (Figure 7E; Delmore et al., 2011). Tumor burden, as determined by bioluminescence, was decreased in (+)-JQ1 treated mice compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 7C-D; Delmore et al., 2011), it was not statistically significant. The opportunity to detect a statistically significant difference was limited though, due to the higher rate of early death in the control group, and increased overall survival in (+)-JQ1 treated mice before the pre-specified tumor burden analysis endpoint. Additionally, we evaluated the (−)-JQ1 enantiomer that is structurally incapable of inhibiting BET bromodomains, which resulted in a minimal impact on MYC transcription, but did not result in a statistically significant difference in tumor burden or survival distributions compared to treatment with (+)-JQ1. Finally, we report meta-analyses for each result.

Replication Study: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs

https://elifesciences.org/content/6/e17584

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper “Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs“ (Sugahara et al., 2010). Here we report the results of those experiments. We found that coadministration with iRGD peptide did not have an impact on permeability of the chemotherapeutic agent doxorubicin (DOX) in a xenograft model of prostate cancer, whereas the original study reported that it increased the penetrance of this cancer drug (Figure 2B; Sugahara et al., 2010). Further, in mice bearing orthotopic 22Rv1 human prostate tumors, we did not find a statistically significant difference in tumor weight for mice treated with DOX and iRGD compared to DOX alone, whereas the original study reported a decrease in tumor weight when DOX was coadministered with iRGD (Figure 2C; Sugahara et al., 2010). In addition, we did not find a statistically significant difference in TUNEL staining in tumor tissue between mice treated with DOX and iRGD compared to DOX alone, while the original study reported an increase in TUNEL positive staining with iRGD coadministration (Figure 2D; Sugahara et al., 2010). Similar to the original study (Supplemental Figure 9A; Sugahara et al., 2010), we did not observe an impact on mouse body weight with DOX and iRGD treatment. Finally, we report meta-analyses for each result. 

Replication Study: Discovery and preclinical validation of drug indications using compendia of public gene expression data

https://elifesciences.org/content/6/e17044

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper “Discovery and Preclinical Validation of Drug Indications Using Compendia of Public Gene Expression Data“ (Sirota et al., 2011). Here we report the results of those experiments. We found that cimetidine treatment in a xenograft model using A549 lung adenocarcinoma cells resulted in decreased tumor volume compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 4C; Sirota et al., 2011), it was not statistically significant. Cimetidine treatment in a xenograft model using ACHN renal cell carcinoma cells did not differ from vehicle control treatment, similar to the original study (Supplemental Figure 1; Sirota et al., 2011). Doxorubicin treatment in a xenograft model using A549 lung adenocarcinoma cells did not result in a statistically significant difference compared to vehicle control despite tumor volume being reduced to levels similar to those reported in the original study (Figure 4C; Sirota et al., 2011). Finally, we report a random effects meta-analysis for each result. These meta-analyses show that the inhibition of A549 derived tumors by cimetidine resulted in a statistically significant effect, as did the inhibition of A549 derived tumors by doxorubicin. The effect of cimetidine on ACHN derived tumors was not statistically significant, as predicted.