Critical Assessment of Alignment Procedures
Data and evaluation scripts
"Critical assessment of alignment procedures for LC-MS proteomics and metabolomics measurements"
BMC Bioinformatics 2008, 9:375
Errata
Supplementary table 3: M1 and M2 labels are inverted in the table.
Proteomics data
The P1 and P2 raw data sets are available at the "Open Proteomics Database".
P1 feature data (featureXML)P1 ground truth data (space-separated)
P2 feature data (featureXML)
P2 ground truth data (space-separated)
Notes on the P1,P2 ground truth file format
Metabolomics data
M1 feature data (featureXML)M1 ground truth data (space-separated)
M2 feature data (featureXML)
M2 ground truth data (space-separated)
Notes on the M1,M2 ground truth file format
Evaluation script
CAAP Evaluation (R-)Script V1.1Notes on the Evaluation Script
Results using eval.R v1.1
Alignment recall and precision results for the proteomics data set P1
| msInspect | mzMine | OpenMS | SpecArray | XAlign | XCMS /wo RC | XCMS /w RC | |
| fraction 00 | |||||||
| recall | 0.5 | 0.81 | 0.86 | 0.61 | 0.82 | 0.72 | 0.62 |
| precision | 0.37 | 0.81 | 0.86 | 0.61 | 0.82 | 0.54 | 0.59 |
| fraction 20 | |||||||
| recall | 0.55 | 0.9 | 0.93 | 0.61 | 0.85 | 0.89 | 0.88 |
| precision | 0.47 | 0.9 | 0.93 | 0.61 | 0.85 | 0.84 | 0.88 |
| fraction 40 | |||||||
| recall | 0.52 | 0.9 | 0.93 | 0.74 | 0.87 | 0.92 | 0.81 |
| precision | 0.41 | 0.9 | 0.93 | 0.74 | 0.87 | 0.85 | 0.79 |
| fraction 60 | |||||||
| recall | 0.58 | 0.84 | 0.96 | 0.71 | 0.87 | 0.91 | 0.77 |
| precision | 0.48 | 0.84 | 0.96 | 0.71 | 0.87 | 0.8 | 0.75 |
| fraction 80 | |||||||
| recall | 0.62 | 0.94 | 0.97 | 0.74 | 0.9 | 0.94 | 0.9 |
| precision | 0.55 | 0.94 | 0.97 | 0.74 | 0.9 | 0.88 | 0.89 |
| fraction 100 | |||||||
| recall | 0.81 | 0.94 | 0.96 | 0.77 | 0.96 | 0.95 | 0.96 |
| precision | 0.58 | 0.94 | 0.96 | 0.77 | 0.96 | 0.89 | 0.96 |
Alignment recall and precision results for the proteomics data set P2
| msInspect | mzMine | OpenMS | SpecArray | XAlign | XCMS /wo RC | XCMS /w RC | |
| fraction 00 | |||||||
| recall | 0.01 | 0.62 | 0.77 | 0.07 | 0.64 | 0.55 | 0.45 |
| precision | 0.01 | 0.49 | 0.65 | 0.05 | 0.51 | 0.32 | 0.39 |
| fraction 20 | |||||||
| recall | 0.47 | 0.87 | 0.92 | 0.57 | 0.68 | 0.88 | 0.87 |
| precision | 0.37 | 0.71 | 0.76 | 0.42 | 0.60 | 0.61 | 0.71 |
| fraction 40 | |||||||
| recall | 0.36 | 0.79 | 0.76 | 0.60 | 0.72 | 0.74 | 0.56 |
| precision | 0.27 | 0.76 | 0.74 | 0.41 | 0.70 | 0.69 | 0.55 |
| fraction 80 | |||||||
| recall | 0.61 | 0.60 | 0.80 | 0.66 | 0.57 | 0.49 | 0.44 |
| precision | 0.38 | 0.56 | 0.70 | 0.45 | 0.56 | 0.42 | 0.43 |
| fraction 100 | |||||||
| recall | 0.64 | 0.80 | 0.90 | 0.63 | 0.83 | 0.80 | 0.76 |
| precision | 0.43 | 0.64 | 0.75 | 0.44 | 0.69 | 0.62 | 0.64 |
Alignment recall and precision results for the metabolomics data sets M1 and M2
| msInspect | mzMine | OpenMS | SpecArray | XAlign | XCMS /wo RC | XCMS /w RC | |
| M1 | |||||||
| Recall | 0.12 | 0.92 | 0.87 | - | 0.88 | 0.97 | 0.94 |
| Precision | 0.68 | 0.73 | 0.69 | - | 0.7 | 0.59 | 0.7 |
| M2 | |||||||
| Recall | 0.12 | 0.98 | 0.93 | - | 0.92 | 0.97 | 0.98 |
| Precision | 0.49 | 0.84 | 0.79 | - | 0.78 | 0.58 | 0.78 |
Comments, questions and new results
rtautenh (at) scripps (dot) edu
sneumann (at) ipb-halle (dot) de
