A cardiotoxicity screen built from ChEMBL bioactivity data — and a demonstration of how much of a machine-learning score in drug discovery can be an artefact of how the data was split.
The same random forest, the same compounds, the same features. The only change is that a scaffold split keeps chemical series whole, so a test compound never has a close analogue sitting in the training set.
That difference is the portion of the apparent performance that was recognising molecules it had effectively already seen. A random split is the default in most tutorials, and on this dataset it overstates the model by that margin. The number a medicinal chemist should care about is the lower one.
—
For a safety screen the asymmetry matters more than the AUROC. Missing a blocker (a false negative) advances a compound that can prolong the QT interval; a false positive only costs you a molecule. A model with high sensitivity and mediocre specificity may be the right one to deploy even when another ties it on AUROC.