In the newly released Mavericks incarnation of iBooks, O‘Reilly ePubs do not render correctly. From my observation, it seems as if subpixel font rendering is disabled (both on default font setting specified by the book, and when using an iBooks default font). Compare the following screenshot:
to one of the same book rendered by Murasaki (an OS X ePub reader):
Although this is an iBooks specific error, it is confined, in my testing, to O’Reilly’s ePubs in my library – non iBooks Store bought ePubs from other publishers render fonts in the quality expected from OS X.
I apologize that it took so long to obtain a response for you. A member of our ebooks team has provided me with some information that hopefully you will find useful. He has provided a suggestion if you're using a computer running Mavericks/ibooks with a non-retina display. What you will please want to do is go to Go to System Preferences > General. Then at the bottom there is a checkbox that says "Use LCD font smoothing when available", and you want to uncheck that box. This appears to make the font smoother. Hope this helps and have a great day!
Customer Service Representative