Miseq vs. Ion Torrent - Round Two.

We have another round of application notes that have been released by both Ion Torrent and Illumina.  This has all the making of a late round heavy weight fight.  First came Illumina with the initial blow (a.k.a application note), comparing the MiSeq E. coli MG1665 data to some of Edge's 314 chips of E. coli DH10B and BGI's 314 chips of the EHEC outbreak strain. I reviewed this application note in the latter part of our previous blog post Ion v. MiSeq – Is There a Competition, And If So, Why? The beginning spent time reviewing the new 316 chip data set released via the Ion Community website.  I focused on drawing comparison between platforms using our own in house analysis of the 316 data set because I thought it drew a more realistic comparison to the platforms.

Post release of the 316 chip, Illumina put out a new application note, now sequencing the DH10B strain itself to ~400X coverage, and highlighting what it obviously sees as deficiencies in the Ion Torrent system.  It spotlights areas such as high predicted QV's of reads, lower numbers of "perfect reads", false positive INDELS and GC representation.  To this end, Illumina released the raw data and an alignment done by their in house software Eland.

In the latest jab back, Ion released a new 314 chip data set using longer read chemistry which allowed for reads to reach > 250 bp and released an application note, highlighting not only the read lengths, but claims of higher actual (not predicted) accuracy, lower SNV rate, and better genome coverage than Miseq, which had almost 40X more data to work with.

I agree with Keith over at Omics!  Omics! Omics! that both sides seem to a certain extent to be talking past each other.  Picking metrics that highlight one's strengths.  Check out his latest blog on the 314 long read data sets and some interpretation.  This segues nicely into a blow by blow comparison of the data sets. First let’s introduce the fighters.  It can get confusing because Illumina is comparing to a 316 data set and Ion is comparing to a new 314 Long Read data set.

So keeping with the corny boxing metaphors - I'll give you the "punch" line first:

Like most things in life, the truth lies somewhere in the middle.

  1. Both Miseq at 400X coverage and 316 Chip at 35X coverage do well in re-sequencing experiments.
  2. Miseq still holds the edge when it comes to de novo assembly, even with the recent release of the 250bp read sets from Ion.

Now, let’s rewind a bit...

Let’s take a look at the ever nebulous "quality scores". Here's what the landscape looks like using predicted QV's

But, Ion suggests that predicted QVs are nice, but when it comes down to it, you want to know what the actual QV's are, and the only way you can do this is by comparing back to a known reference.  AGREED.   Ion goes on to claim that Illumina overcalls by some 5-10 phred points (big numbers), looking at the plot versus predicted phred scores on raw Ion data, it's obvious they are under calling (as pointed out in Keith’s post in great detail).  Check out fig.2 in the application note.  Flipping it around however, this is not accounting for INDEL errors (how would you measure something that doesn't exist?), so only a partial picture of overall accuracy.  This applies for both 314 and 316 chips.

Here's the mismatch by location histogram for Illumina

and the one for ion torrent (note this is the 316, 100bp data set)

I think this brief example shows how you can get lost in the bushes when trying to evaluate platform via application notes and interpretations from the vendors throwing all kinds of numbers, sometimes past each other.  Another complicating factor is the (proper) usage of tools that work best on the application at hand.  For example, as pointed out in the Ion application note, the MiSeq data only represents 94% of the genome.  In reality (using multiple tools*) I was able to generate a more accurate alignment that covered 100% of the genome and reduced the 13 false positive substitution errors to only 1. I'm not sure if it is because Eland is tuned toward large mammalian genomes (I know the Life Tech Bioscope default pipeline parameter are - and this makes perfect sense), or it was end user mis-hap, dunno.  On the flip side, Illumina highlights over 2500 false positive indel calls using the alignment from TMAP on the 316 chip, and SNP/INDEL calling using mpileup.  However, neither TMAP or mpileup are necessarily well tuned for microbial Ion data yet.  Using the same pipelines from the Illumina data set, I found on the 316 Chip that one could cut the INDELS to less than 50 with one pipeline and less than 10 on the other - the first coming out of a better alignment and software* suited to accounting for homopolymer issues, the latter using a basic, high allele frequency cutoff to filter het INDELS in a haploid organism.

One thing the Illumina application note touches on is the concept of the perfect read.  Esoteric metric in my book, but OK, let’s play along.  Illumina claims upward of 90% of the reads to be error free.  I wonder if this is of the reads that mapped only, because if so, I could easily make it 100% (wink wink).  This could be skewed by analysis software and how it is run (are you detecting a theme here?).  Or maybe those reads with errors are in the 6% of the genome that Eland failed to align to.  They show Ion clocking in at around 60%.   In looking at both data sets through a single pipeline we looked at the number of reads that are mapped to the reference genome with no difference from the reference genome, only counting reads that are mapped confidently (Read has a mapping with ≥ 90% probability, and can be considered equivalent to “unique”).

Illumina = 9,790,519 ( 57.02%)

Ion = 732,625 ( 43.42%)

A bit less dramatic...

So, on to GC bias.  We looked at the average coverage calculated per GC content bucket. More specifically, for every possible GC content value (from 0% to 100%) on the horizontal axis, the vertical axis shows the average coverage of all 100bp genomic windows that have that GC content.  Looks pretty similar to me.  If anything, looks like Illumina maybe over representing content that is < 20% GC (see graph below). My guess is that the difference in tools would account for dropping high or low GC areas as well.


Ion Torrent

Another plot shows Illumina favors AT

and Ion favors GC rich regions

Again - a bit less dramatic...

Clouding the water somewhat is Ion releasing a new data set from an older 314 chip with longer read chemistry.  Ion draws most of the comparisons to Miseq using this data set.  While Illumina focused on the 316 chip.  (Are you still following along?)  In my opinion, it should have used the higher quality 316 data set to draw comparison on quality, and let the long read stuff stand on its own.  It’s absolutely amazing at the speed in which Ion is releasing new features, and long reads in the end is what I think they will be able to hang their hat on, but this data set falls short on quality, which isn't necessarily a bad thing, unless your tech note is called "The Ion PGM™ sequencer exhibits superior long-read accuracy".

In our analysis, the 314 LR data set under-competed against the 316 data set in both resequencing and de novo applications (more later on de novo).  The inferred error rate across the board for the LR 314 chip is around 3%, a smidge higher than what we saw for the traditional 314 chips and almost double than the 316 chip.  In turn, we saw around 40x more FP INDEL calls during our WGS resequencing applications. So, for the long term success of the platform, this is great news, newer chips have less sequencing error than the older ones, and homo-polymer issues and FP INDELS as a result have been reduced by an order of magnitude, and Ion promises a similar return over the coming six months.  The confusing part is presenting a data set in a light that claims high accuracy, and it isn't the most accurate data set you have in your pocket.  My feeling is that they really wanted to highlight accuracy in the 100-150 bp compared to Miseq, which in looking at actual quality values, they have a clear advantage (only looking at substitution errors though). 

I think it got lost in the noise though. I think de novo assembly is a great place to showcase your long reads, and honestly when I heard a set of reads pushing 250+ was available, that was the first thing that popped into my head.  Nick Loman has already taken a look at  Ion Torrent and the impact of the new longer reads on assembly? Long story short, it doesn't help.  We see that too.  Even after accounting for potential coverage issues (316 runs had more data).  But, as we highlighted before, the de novo assembly from the 316 chip, while not as "good" statistically as the Illumina assembly at similar coverage, I still think that given the price point, it’s a compelling alternative for many reasons (you'll have to read the other blog post to see why I think that).  SO, if one could create the perfect super hero from the 314 long read data and the accuracy of the 316 chips, there’s no telling what super de novo powers it could have.  Nick also highlights that what is left un-assembled in the genome is most likely repetitive content, and throwing more reads at it won't help, so this makes a great case for the future of Ion, whereas Illumina may hold the edge for now in throughput, there are some things that just throwing reads at won't solve (repeats for one), and to date (after many requests), I still can't get anyone to convince me that Illumina hasn't hit the ceiling in read length.

The LR 314 chip run is something Ion should be proud of, and as we have seen, they have the ability to make marked improvements on accuracy and throughput over time, so to add read length to the repertoire gives this platform a 3 pronged attack plan going forward.

Another idea that falls out of this analysis is that choosing your analysis software can have almost as much affect as choosing your platform to do the sequencing.  Illumina deserves the rocking they got from Ion, putting a data set out with such a low quality alignment, but if you take some time to look closer, you see that almost all of what Ion highlights as deficiencies of the Miseq platform is mitigated (Substitution errors, genome coverage, etc) and almost all of what Illumina highlights as deficiencies of the Ion platform is mitigated (INDEL errors, predicted QVs, GC bias). I do want to highlight that even though we could produce an alignment with 100% coverage, this is almost besides the point, because most people will use Eland and other vendor tools to do the analysis - so in theory, it was covered 100%, in reality, it was only covered 94% for the researcher that ran Eland.  Here is where I give another Edge (wink wink) to Ion - following an open source policy on software.  Bringing on the likes of Nils Homer (of BFAST and SeqAnsers fame) shows they have their heads on right.  So, choose your tools and analysis people as careful as you choose your sequencing platforms.

*CLCBio genomics Workbench and DNANexus were used to do whole genome re-sequencing analysis on all data sets.  Both pipelines (validated internally) are well tuned toward microbial data sets.  Using a single pipeline reduces variability between analysis of data sets, and replicating with an alternative pipeline able to analyze all data sets give us the warm and fuzzy that we did it right.  Denovo assembly was carried out using Newbler for the Ion Torrent data.  To date, we have used CLC, Celera Assembler, MIRA, Velvet, and Newbler. Newbler has produced the best assembly statistics and most accurate assemblies to date.

Link to the DNANexus alignment and variant analysis for the Miseq data

Link to the DNANexus alignment and variant analysis for the Ion data

You will need a free DNANexus account to view this data.