This is great example of the type of histogram on which the Fenyo method of calculating E values performs poorly.

See that one tall histogram bar? As that falls in the small section where the least-square line is calculated what happens is a much steeper slope (and, directly, a much better E value) comes out than really should be.

These kinds of histograms happen all the time in databases which contain peptides derived from multiple splicing junctions. The reason is that there may be many peptides where the front end or tail end are correct. For example, let’s say the correct peptide for a spectrum is “WSFFFFCGYN”, but an intron position begins after the C. The database may contain many variations on that peptide which begin with “WSFFFFC”. If those alternate peptides also fall within our precursor tolerance then they will all produce fairly decent scores, which will create the kind of spike such as that in the above histogram.

How can we measure this?

Relatively easy. The somewhat challenging part is getting a set of spectra from your machine where you know the peptide from which each spectrum came. These tests sets are very useful and finding consistent theoretical peak error is one of them.

To calculate average theoretical peak error, go through each spectrum you have. For each spectrum, take the peptide it was derived from and find the theoretical ions of this peptide. Then, for each of these theoretical ions, find the most intense (it’s so intense!) peak of the spectrum that falls within a fairly conservative mass window. (This mass window should be large enough to account for normal error but small enough to avoid false matches. A good starting point is 0.5Da.) Find the mass difference between this peak and the theoretical ion.

Do this for every theoretical ion of every peptide for every spectrum and get the average.

Voila!

The animation!

and spectrum:
T10475_Well_A13_2025.07_17096.mgf..pkl

are the same. This is bad news for me as TandemFit gets both of those “wrong”. The quotes because TandemFit’s match of QAGLQLQESLEPAVRLDR has 11 fragment alignments vs 2 produced by VPAPSIEDICHVLSTVCK which is the “correct” peptide.

Update: other duplicates:
T10475_Well_A12_1386.68_16898.mgf..pkl
T10475_Well_A12_1386.68_17096.mgf..pkl

T10475_Well_A03_1551.77_16898.mgf..pkl
T10475_Well_A03_1551.77_17096.mgf..pkl

T10475_Well_A11_1386.69_16898.mgf..pkl
T10475_Well_A11_1386.69_17096.mgf..pkl

T10475_Well_A10_1188.45_17096.mgf..pkl
T10475_Well_A10_1188.45_16898.mgf..pkl

T10475_Well_A10_2143.98_17096.mgf..pkl
T10475_Well_A10_2143.98_16898.mgf..pkl

Hmm… it looks like something of a pattern

Histogram for Spectrum A:

E value: 1.0659386771035567E-26

Histogram for Spectrum B:

E value: 2.64817948989741E-19

These two score distributions typify what you’d expect when a correct match is present. Most scores fall into a large mass on the low end, but then, on the right most side, far away from the other scores, you see a tiny blip where the correct score lives.

Now, just by looking at the two would you say that you are much more confident about the results of the first one over the second one? Maybe so, but would you say that you are 10,000,000 times more confident? Well, that’s what those E values imply. Like I said before, E values are highly sensitive to the shape of the right tail of the histogram, and the slight difference in shapes between the two skewed the linear regression to make the confidence factors so starkly different.

This dependence on histogram shape can be used to game the E value method to make a scoring algorithm that produces results with highly overly inflated confidence scores.

What I’m about to do is simple and can be implemented by any scoring method.

IMPORTANT NOTE: I’m not changing the E value code at all. I’m only changing the code in the scoring method. I’ve gone ahead and altered the code in TandemFit to square the final score. Here’s the code:

score = score * score;

Simple enough? Do you think this improves the quality of the scores?

Spoiler: It doesn’t. This will not change the ordering of TandemFit scores. The best score will still be the best score and the worst score will still be the worst scores. However, the distribution of the scores will be very different. What were scores of 1, 2, 3, 4 and 5 will now be scores of 1, 4, 9, 16 and 25. Because larger scores are more affected by the squaring process, this will appear to make our best scores (the true matches) seem much further removed from the rest.

With this one change, here are the histograms and E values for the scores for those same two spectra:

Histogram for Spectrum A:

E value: 3.202345672236334E-91

Histogram for Spectrum B:

E value: 1.1669794248500472E-57

Look at those frickin’ E values! The first one is within throwing distance of being one over a googol!

Now let’s say we were comparing the two scoring methods. For spectrum A, is our confidence really 10^65 better for the second scoring method?

Will you ever trust anyone’s E values again?

I’ve created a test set of spectrum/peptide pairs which I think does a good job of filtering the impurity peptides. It was a much smaller percentage than I had originally guessed — Only about 1434 spectra out of 7036 come from peptides in the USP set.

About the creation:

The first pass at creating this body of spectrum/peptide pairs was created by taking all matches that were in the top 10 results for TandemFit that were also in the USP peptide list. I then went through and “hand verified” the results. This was done by printing out and evaluating a report of all of the spectra/peptide matches that were incorrectly assigned. This included graphic representations of the spectra with peaks hilighted for aligned theoretical ions. I went through this report and selected out spectra where the TandemFit match seemed to far outclass the match from the USP set. This amounted to removing 5 objectionable spectra. These wonky results may have been produced by spectra chimera or by odd luck that one of the peptides from the USP list had similar properties to one of the impurity peptides.

Of course, I recognize that this “hand verification” may further bias the dataset. You can look at the ones that I have removed here to see if I’m being crazy:

http://proteomics.me/files/20100628

Granted, as the test set was created using TandemFit, it will be somewhat biased towards TandemFit. However, it is very good for potentially tuning some of the parameters in the algorithm to get the matches that were in positions 2 through 10 into position 1.

As far as how many hits we have missed by going with the top 10 matches, I went through the same process of selecting and hand verifying with top 5 matches. Going from the top 5 to the top 10 means the match set went from 1340 to 1434. That’s a 7 percent gain. If I repeated the process for the top 20 matches I wouldn’t expect more than another 7 percent gain. I would bet that this test set accounts for at least 90% of the correct matches.

Results from these tests.

Again, because of the above assumptions, these numbers are approximate.

Percent correct at one percent error rate: 57.53%
e value at that rate: 0.1349
Total correct top matches: 82.5%

The name works well in that it scores tandem mass spectrometry data, and it does so by sequentially comparing spectra and theoretical spectra in tandem.

This can be handy for SpectrumMatch as well as for comparing scores TandemFit scores (formerly MSMSFit).

Peppy, the open-source proteogenomic mapping software has been benchmarked at processing over 600,000 spectra per day on a consumer-grade desktop…