What I would like O'Reilly to do to fix these problems is to modify the program used to generate the ePub files such that (1) the NCX TOC is flat, containing only the Parts and Chapters of a book and not the sections of each chapter and (2) the ePub files contain a real, "printed" TOC that contains all the details currently included in the NCX TOC. In this way, those of us who wish to dig into our favorite O'Reilly titles can do so without having to wait inordinate amounts of time to navigate to a chapter or page in a book.
Full details of Research and Experimentation
I did some experimentation on my end to try and isolate the cause of the problem, first trying another O'Reilly Book (C# 4.0 in a Nutshell [3MB file size], Ch. 12; waited 20--30 seconds for the chapter to load), a couple of books published by Informit (Code Reading: An open source perspective [9.3MB file size] & Domain-Specific Languages [27MB file size]; average seek time of 3--5 seconds in each book). I also referenced a (Urp!) DRM-protected book published by Wrox called "Microsoft SQL Server 2008 Bible" (34 MB file size) and experienced load times of 3--5 seconds.
Therefore, the cause of the extreme load times for O'Reilly books on my eReader is the size of the NCX TOC files O'Reilly includes in their eBooks.
Help get this topic noticed by sharing it on Twitter, Facebook, or email.
For some context on our current EPUB specs, several years ago we made a decision to exclude a "printed" version of the book's Table of Contents in the main book flow, and to solely use the NCX file to communicate TOC data. There were two main reasons we settled on this approach:
1. The NCX file is the canonical mechanism in the EPUB 2.0.1 specification for communicating Table of Contents data to the ereader, and all major ereaders have a built-in UI for navigating the book via the NCX TOC. As such, we felt having an additional HTML version of the TOC in the book content was redundant.
2. Not only was having the additional HTML version of the TOC redundant, but it was also potentially an annoyance to the reader, as he/she would have to flip through these extra TOC pages to get to the beginning of the actual book content.
So that's the rationale behind our present TOC policies for EPUB. However, the most fundamental precept of O'Reilly's ebook development group is that we continuously strive to make our ebooks the best experience possible for all our customers, and we are willing to revisit pretty much any of our existing policies when new data comes to light that militates against them. Your findings on how a large NCX file degrades the reader experience on Kobo Glo certainly make a compelling case that we should revisit our TOC specs. So we are going to do just that and look into how we can better optimize the file size of our NCX documents, as well as reconsider inclusion of an additional HTML TOC in the main content flow.
I can't promise that we will ultimately implement your exact suggestion, because when optimizing our EPUB content, we need to do so in light of the full ecosystem of ereaders that our customers use to consume our ebooks, which includes dozens of different hardware devices and software platforms that can vary quite a bit in terms of technical specs and rendering capabilities. What works best for Kobo Glo may or may not work well on NOOK Simple Touch, and we do our best to take tradeoffs like this into account when designing our EPUBs to ensure that our content is as device-agnostic as possible. But what I can promise is that as we continue to iterate and improve our EPUB-generation toolchain, we are going to make TOC optimization a top priority.
Finally, I just want to add, on a more general note, that O'Reilly welcomes and encourages its customers to "hack" their EPUB files if there are customizations they want to make, just as you've done using Sigil. We make all O'Reilly ebooks available for purchase completely DRM-free, because we believe that when you buy an ebook from us, you own that content and should feel free to tweak it however you see fit for your personal use.
Please let us know if you have any further questions about our EPUB specs or policies, or if you have any further feedback about your ebook reading experience on Kobo Glo or other ereaders. Again, we greatly appreciate all your feedback.
Publishing Technology Engineer
O'Reilly Media Inc.