Thursday, July 19, 2012

Koboish ebooks

I really like the kobo touch but I noticed that there is a difference in how side-loaded epubs and epubs downloaded from the kobo store are treated by the device. The thing that I liked most about the ones from kobo was the page numbering that corresponds to actual screens required to reach the end of a chapter. After noticing this I started combing the net and discovered the infamous kepub format. On finding this I also found that changing the extension of the files from .epub to to .kepub.epub should give me access to some of those features so I did just that.
Results were mixed:
  1. Hooray, we've got page numbering
  2. In addition to page numbering we can also control the text appearance  using the fonts menu
  3. hmmm, where did the covers go? (actually I did read about this happening)
  4.  the third thing is more subtle and I only noticed that after reading about 30 pages of a chapter, closing the book and then trying to pick up where I had left off: while the chapter was correct, it always loaded the first page. The problem here is that I could no longer add annotations to the book.
Since I really, really wanted that page numbering but I don't have the time to read a book in one go I started to poke around a little.
A warning first:
  1. this work on the kobo touch with the 2.0 firmware
  2. While I've managed to get covers and annotations working I don't know if anything else broke
  3. FOLLOW THIS GUIDE AT YOUR OWN RISK
The first thing you need to do is to change the extension of your epub to .kepub.epub and upload it to your kobo (disconnect the device to trigger it to detect the book). The rest I'll split into three parts:
Give me my cover back
Recover annotations
Get lazy with scripting


Give me my cover back

 The first thing I wanted to do was get rid of the ugly kobo generated covers. Even if the epub has a cover set after changing the extension to .kepub.epub the cover is no longer used and is replaced with some generated covers. 
In order to fix this you need to do the following:
  1. Assuming you cover is a jpeg file named mywonderfullcover.jpg
  2. you will need your cover in four sizes (you can also convert it to grayscale to reduce the size)  and each one of this files needs to have a special name and the extension changed from .jpg to .parsed:
    1.  "mywonderfullcover  - N3_LIBRARY_FULL.parsed" - size: 355 x 530 pixels
    2.  "mywonderfullcover  - N3_LIBRARY_GRID.parsed" - size: 149 x 223 pixels
    3.  "mywonderfullcover  - N3_LIBRARY_LIST.parsed" - size: 60 x 90 pixels
    4.  "mywonderfullcover  - N3_LIBRARY_SHELF.parsed" - size: 40 x 60 pixels
  3. Connect you kobo and copy the files you created to the .kobo/images folder 
  4. Next you'll need to update the sqlite database. The file can be found at .kobo/KoboReader.sqlite. I suggest making a back of this first. Open the database. I use SQLiteSpy but you can use any sqlite browser you like. Open up the content table and look for the row that contains your book name in the ContentID column and has the ContentType column set to 6. The ImageId column for this row should be empty. Set it to "mywonderfullcover".
  5. Save the changes and disconnect the reader
  6. In order for the cover to be detected you have to restart the reader (if anyone knows of another way of refreshing the data from the database please let me know) 
Now you should have your cover back, plus the additional benefits already listed but you still don't have any annotations


Recover annotations

At first I thought that the annotations are kept somewhere in the database...I was wrong.
I also tried putting the file in the .kobo/kepub folder and adding the required entried manually in the database but still there were no annotations  although I did learn some stuff about the database structure while doing this
After doing a little more reading on the forums I found some opinions that there are some special javascripts involved so I opened a free book that I got from the kobo site. This also proved wrong but it did lead me in the correct direction.
There doesn't seem to be any javascript involved but, in looking at the html files it look like the html files inside the kepubs have some special tags inserted that allow annotations to be retrieved. 
So what I did was open each html file and wrap the content of each top level h and p tags in a span tag with the id set to "kobo.[incremental_count]".1 where incrementa_count starts at 1 and goes as high as you need it to.
To illustrate this let's suppose you have the following insisde the body tag of your html:
<h1>Chapter</h1>
<h2>SubChapter</h2>
<p>some paragraph</p>
<p>some other paragraph</p>
<p>the last paragraph</p>

This would be changed to:
<h1><span id="kobo.1.1">Chapter</span></h1>
<h2><span id="kobo.2.1">SubChapter</span></h2>
<p><span id="kobo.3.1">some paragraph</span></p>
<p><span id="kobo.4.1">some other paragraph</span></p>
<p><span id="kobo.5.1">the last paragraph</span></p>

Upload your .kepub.epub back on the device and enjoy the freshly recovered annotations.


Get lazy with scripting

 

Automatic renaming using calibre

If you use calibre you can configure it to rename your file when uploading it to the reader by going to "Preferences->Sending books to device" and  and adding ".kepub" at the end of the title template.


Automatic photo shop script for updating images

If you use photoshop you can use the script below to automatically create and rename your cover files (save the code in a file with the extension set to .jsx). Just open your jpeg file in photoshop and go to  File->Scripts->Browse and select the file where you saved the script code. Your original file will be left unchanged.
    #target photoshop
    main();
    function main(){
    if(!documents.length) return;
    var startRulerUnits = app.preferences.rulerUnits;
    app.preferences.rulerUnits = Units.PIXELS;
 var quality = 10
    var doc = app.activeDocument;
    var Name = doc.name.replace(/\.[^\.]+$/, '');
    var Path = decodeURI(doc.path);
 var outFolder = Folder(Path);// +"/"+Name);
    if(!outFolder.exists) outFolder.create();

 doc.changeMode(ChangeMode.GRAYSCALE);
 doc.bitsPerChannel = BitsPerChannelType.EIGHT
    createNamedSnapshot("Snap 1");

    doc.resizeImage(355, 530, 96, ResampleMethod.BICUBIC);
    var saveFile = File(outFolder +"/"+Name+" - N3_LIBRARY_FULL.parsed");
    SaveJPEG(saveFile,quality);
    var saveFile = File(outFolder +"/"+Name+" - N3_LIBRARY_FULL.jpeg");
    SaveJPEG(saveFile,quality);
    revertNamedSnapshot("Snap 1");
 
    doc.resizeImage(149, 223, 96, ResampleMethod.BICUBIC);
    var saveFile = File(outFolder +"/"+Name+" - N3_LIBRARY_GRID.parsed");
    SaveJPEG(saveFile,quality);
    revertNamedSnapshot("Snap 1");
 
    doc.resizeImage(60, 90, 96, ResampleMethod.BICUBIC);
    var saveFile = File(outFolder +"/"+Name+" - N3_LIBRARY_LIST.parsed");
    SaveJPEG(saveFile,quality);
    revertNamedSnapshot("Snap 1");
 
    doc.resizeImage(40, 60, 96, ResampleMethod.BICUBIC);
    var saveFile = File(outFolder +"/"+Name+" - N3_LIBRARY_SHELF.parsed");
    SaveJPEG(saveFile,quality);
 
    app.activeDocument.close(SaveOptions.DONOTSAVECHANGES);
    app.preferences.rulerUnits = startRulerUnits;
    }

    function SaveJPEG(saveFile, jpegQuality){
    jpgSaveOptions = new JPEGSaveOptions();
    jpgSaveOptions.embedColorProfile = true;
    jpgSaveOptions.formatOptions = FormatOptions.STANDARDBASELINE;
    jpgSaveOptions.matte = MatteType.NONE;
    jpgSaveOptions.quality = jpegQuality; //1-12
    activeDocument.saveAs(saveFile, jpgSaveOptions, true,Extension.LOWERCASE);
    }

    function createNamedSnapshot(name) {
        var desc = new ActionDescriptor();
            var ref = new ActionReference();
            ref.putClass( charIDToTypeID('SnpS') );
        desc.putReference( charIDToTypeID('null'), ref );
            var ref1 = new ActionReference();
            ref1.putProperty( charIDToTypeID('HstS'), charIDToTypeID('CrnH') );
        desc.putReference( charIDToTypeID('From'), ref1 );
        desc.putString( charIDToTypeID('Nm  '), name );
        desc.putEnumerated( charIDToTypeID('Usng'), charIDToTypeID('HstS'), charIDToTypeID('FllD') );
        executeAction( charIDToTypeID('Mk  '), desc, DialogModes.NO );
    }

    function revertNamedSnapshot(name) {
        var desc = new ActionDescriptor();
            var ref = new ActionReference();
            ref.putName( charIDToTypeID('SnpS'), name );
        desc.putReference( charIDToTypeID('null'), ref );
        executeAction( charIDToTypeID('slct'), desc, DialogModes.NO );
    }

 

Automatic kobofying

Now all that is fine but what if you don't want to spend twice as much time as is requred to read a book just to kobofy it.
No worries, since I'm a little lazy myself I create the small python script bellow. In order to run it you need python 2.7 and BeautifulSoup installed. 
In order to use it unpack your files to a folder, create file called something.py inside that folder containing the code bellow and run it by calling "python something.py". 
Warning: this script was hastily put toghether so it has a few restrictions:
  1. It always adds the spans to h and p tags so don't run it multple times on the same file 
  2. It only works on top level h and p tags so, for example, if your h and p tags are wrapped in a div or another tag it won't work.

import sys
from bs4 import BeautifulSoup
import re
import os, os.path
 
def altertags(soup):
 counter = 1
 for tag in soup.body.find_all(re.compile("^(p|h)"), recursive=False):
  new_tag = soup.new_tag("span", id="kobo."+str(counter)+".1")
  counter = counter + 1
  tag.wrap(new_tag)
  tag.unwrap()
  new_tag.wrap(tag)

def parsefile(infile, outfile):
 soup = BeautifulSoup(open(infile))
 altertags(soup)
 output = open(outfile,'w')
 output.write(str(soup))
 output.close()
 

if (len(sys.argv) > 2):
 infile = sys.argv[1]
 outfile = sys.argv[2]
 parsefile(infile, outfile)
 
if (len(sys.argv) == 2):
 infile = sys.argv[1]
 parsefile(infile, infile)

if (len(sys.argv) < 2):
 for file in os.listdir("."):
  if file.endswith("html"):
   parsefile(file, file) 
If someone can improve this please share.
I don't actually use this as it requires you to unpack the book, repack and so on and that's to much work for me so I just modified kiwidude's Modify Epub plugin for calibre to do this for me.

Well...that's about it. If anyone reads this and finds it useful feedback is always appreciated.