Development of calibre

  1. Development of calibre
    1. Design philosophy
    2. Adding support for new Input/Ouput formats
    3. Adding support for new devices
    4. Browsing the source code
    5. Setting up the development environment
    6. Instructions for linux
    7. Instructions for Windows
    8. Instructions for OS X
  2. Sony USB Protocol
  3. Translations
    1. Translations FAQ
  4. Documentation
  5. Videos
  6. Contributors
    1. Contributors of code/bug fixes
    2. Graphics
    3. Recipe writers
    4. Other projects whose code/libraries calibre uses/links against
    5. Translators

Design philosophy

I come from the Unix world, which means calibre is architected in little pieces each with its own command line interface which means that each piece of functionality is reusabe not just in python programs but in any software. In fact adding complete support for converting any new ebook format to calibre requires the writing of only two converters format->html and html->format as well as a metadata reading/writing tool. All the other features of calibre will work automatically with these three converters in place. Similarly, writing a device driver requires very little work. You basically have to implement a well-defined interface (interface.py). Once you do that, your device will have full support in calibre.

Adding support for new Input/Ouput formats

calibre is written primarily in Python with a few C extension modules. So you have to contribute plugins in either Python or C/C++. Because of its modular design and support infrastructure, adding suport for new formats is simple:

  • A new input format
    • A module that can read metadata from files in the input format.
    • A module that can convert the input format into one or more HTML files.
  • A new output format
    • A module that can write metadata to files in the output format.
    • A module that can convert a sequential list of HTML files into a file in the output format.

If you create a plugin to do this, just let me know, and I will take care of integrating it into the calibre code base.

Adding support for new devices

You have to create a device driver that implements the DeviceInterface defined in interface.py. You should be able to lift a lot of code from the existing drivers in the calibre.devices package.

Browsing the source code

The source code can be browsed online in launchpad. If you're interested in development, you can submit patches as outlined below. Also consider joining the calibre-devs team and subscribing to its mailling list.

Setting up the development environment

If you are on linux, setting up a development environment is very simple and is outlined below. If not, there is a VMWare Appliance with a fully setup development environment calibre-gentoo.7z. Passwords are the same as usernames. Once you have it running, log in as kovid and start up Eclipse, it has the calibre project already setup. Update the code from bzr and run make as shown below. Submit patches and if you want to do a lot of development talk to me about getting commit access to the bzr repository.

Instructions for linux

If you want to submit patches or follow development, you can check out and use calibre from bzr with the commands 

bzr branch http://bzr.kovidgoyal.net/code/calibre/trunk calibre
cd calibre && python setup.py build && sudo python setup.py develop

On windows and OSX you may get a Memory Error while trying to do the above. In which case locate the file socket.py in your python installation and change line 308 to 

recv_size = min(65636, max(self._rbufsize, left))

To update from bzr 

python setup.py develop --uninstall
bzr merge && make && sudo python setup.py develop;

To create a patch to submit for inclusion: 

bzr commit -m "Describe your change here"
bzr merge
bzr send -o my-change

Attach the file my-change to a ticket with a short description of the change and why you think it should be included.

Instructions for Windows

Windows Development page.

Instructions for OS X

OSXDevelopment page.

Sony USB Protocol

The SONY USB protocol is documented via the API documentation in HTML and PDF. A good way to start understanding the protocol is to run prs500 with the --log-packets option. This will give you a packet trace of communication between the device and host. Each packet is identified in the log. The packet format is documented in the API documentation of calibre.prstypes.

If you want write access to the wiki please register and then contact me: calibre at kovidgoyal.net.

Translations

calibre has support for translating its user interface. You can see the list of languages for which translations exist here. If the translations for your language are incomplete, you can contribute the translations using a simple web-based interface. To do so first register/Log in at Launchpad, then click on your language in this page. You can filter the results to show only untranslated strings, and contribute translations for them.
It is more comfortable to download the PO files and edit the PO file you want to update. Click on the corresponding language here and upload the PO file as "User upload". You have to be logged in to upload files.

If you want to add support for a new language, add a comment to #283 and I'll add it to Launchpad.

Translations FAQ

  • What to do with the & symbols?
    • The & symbols denote a keyboard shortcut. So for example the string "&Title" means that the corresponding GUI element can be accessed with the keyboard shortcut "Alt+t" (this only works on windows/linux). SO in your translation leave the & symbol before the same letter if possible, or some other suitable letter.
  • What to do with the "%A_Word" characters?
    • The % character and a word to the right of it (no space between the percentage sign and the word), denotes a word that will be replaced by the code. So when encountering this symbol and a word right next to it (example: %default) then don't change it. Keep it in the translated sentence.
  • How do I test my translations?
    • First you need to download a .po file for your language from here. You can test your .po file by running calibre from the command line from the directory in which the .po file resides. This is only tested in Linux. On windows also it should work. On OSX, since you can't launch the GUI from the commandline, try putting the .po file in your home directory.

Documentation

The documentation for calibre is in the form of a User Manual. The User Manual is automatically generated from txt files in the Sphinx documentation format. The source files are located at manual. You can add to the documentation and submit patches for inclusion.

Videos

Instructions for making demo video on linux, using the utility recordmydesktop. Recording is done with the command 

recordmydesktop -windowid $(xwininfo | awk '/Window id:/ {print $4}') -device hw:1,0 -delay 2 --overwrite  -o out.ogv --full-shots

Click on the window to record and recording will start in 2 secs. hw:1,0 is the sound recording alsa device (obtained using arecord -l). The video file out.ogv is in theora format. It can be converted to MPEG4 and FLV as shown below. For avi: 

mencoder out.ogv -ovc lavc -lavcopts vcodec=mpeg4:vpass=1 -oac ac3 -o out.avi
mencoder out.ogv -ovc lavc -lavcopts vcodec=mpeg4:vpass=2 -oac ac3 -o out.avi

For Flash video 

mencoder out.ogv -o out.flv -of lavf -ovc lavc -oac mp3lame -lavcopts vcodec=flv:vbitrate=250:autoaspect:mbd=2:mv0:trell:v4mv:cbp:last_pred=3:predia=2:dia=2:precmp=2:cmp=2:subcmp=2:preme=2:turbo  -srate 22050 -af lavcresample=22050

You can add -vf scale=320:240 for standard flash video resolution.

Contributors

calibre is an open source app that uses a large number of other open source apps and is developed on Linux, an open source platform, as such, it has benefited from a the work of large numbers of people, some of whom are listed below. I've tried to be as exhaustive as possible, but, if I have forgotten your contribution, please let me know.

Contributors of code/bug fixes

In no particular order:

  • llasram: For epub2lrf, lit2oeb and various improvements in the html2lrf code
  • Roger Critchlow: For the initial implementation of lrs2lrf
  • Anatoly Shipitsin: For fb2lrf and fb2-meta
  • Ori Avtalion for various bug fixes
  • Marielle for the Kindle driver
  • Igor Skochinsky (for various bits of reverse engineering without which calibre would never have gotten off the ground)

Graphics

  • diaboNL for the reader icon
  • Caleb Avery for the logo
  • The Oxygen project for various icons

Recipe writers

Too many people have contributed recipes to list here, but their contributions are a big part of calibre.

Other projects whose code/libraries calibre uses/links against

  • Python
  • Qt
  • PyQt
  • rtf2xml
  • pdftohtml
  • convertlit
  • pdftoxml
  • feedparser
  • pylrs
  • BeautifulSoup
  • py2exe
  • py2app
  • mechanize
  • pictureflow
  • sqlite
  • chardet
  • pypdf
  • imagemagick
  • Python Imaging Library
  • Sphinx
  • NSIS

Translators

The list of translators is available at https://translations.edge.launchpad.net/+groups/calibre

Last, but not least I'd like to thank the folks at http://mobileread.com for creating a wonderful community for us e-book afficionados to gather round.