# Leakage audit

Every accuracy number in this repository assumes the compounds being scored are independent of the knowledge scoring them. This checks that assumption instead of making it. **The output is not a score — it is how much to discount every other score here.**

| Dataset | Compounds | Identity leaks | Shares a framework | Verdict |
| --- | --- | --- | --- | --- |
| **EGFR** | 4604 | 14 (0.30%) | 451 (9.8%) | identity leakage present but small |
| **PARP1** | 4574 | 5 (0.11%) | 28 (0.6%) | identity leakage present but small |
| **PRMT5** | 1062 | 0 (0.00%) | 0 (0.0%) | no identity leakage detected |

### EGFR: structures that leaked

| Compound | Role in the knowledge base | Measured pChEMBL |
| --- | --- | --- |
| afatinib | covalent geometry reference (EGFR) | 9.24 |
| neratinib | covalent geometry reference (ERBB2) | 8.96 |
| lapatinib | covalent geometry reference (ERBB2) | 8.41 |
| osimertinib | covalent geometry reference (EGFR) | 8.35 |
| gefitinib | covalent geometry reference (EGFR) | 7.96 |
| erlotinib | covalent geometry reference (EGFR) | 7.72 |
| lorlatinib | covalent geometry reference (ALK) | 6.875 |
| midostaurin | covalent geometry reference (FLT3) | 5.795 |
| tucatinib | covalent geometry reference (ERBB2) | 5.6 |
| crizotinib | covalent geometry reference (MET) | 5.57 |
| gilteritinib | covalent geometry reference (FLT3) | 5.46 |
| tamoxifen | covalent geometry reference (ESR1) | 5.45 |

### PARP1: structures that leaked

| Compound | Role in the knowledge base | Measured pChEMBL |
| --- | --- | --- |
| talazoparib | covalent geometry reference (PARP1) | 9.185 |
| rucaparib | covalent geometry reference (PARP1) | 8.7 |
| palbociclib | covalent geometry reference (CDK4) | 8.46 |
| olaparib | covalent geometry reference (PARP1) | 8.3 |
| niraparib | covalent geometry reference (PARP1) | 8.04 |

- **EGFR**: 14 of 4604 compounds (0.30%) are reference structures. Too few to move an AUC, and they should still be excluded before any claim about prospective accuracy.
- **PARP1**: 5 of 4574 compounds (0.11%) are reference structures. Too few to move an AUC, and they should still be excluded before any claim about prospective accuracy.
- **PRMT5**: no benchmark compound is a reference structure; 0.0% share a framework with one.