Kindle and Calibre - Creating EBooks

Something Calibre does very well is convert ebooks from one format to another. And because of that, I've been able to do something that I simply couldn't have imagined doing just one year ago. I've created a couple of ebooks. From scratch.

I'm not going to be sharing these books, sorry, but I can share how and why I made them.

The why is pretty simple, really. I'm going to be headed to Emerald City Comicon this weekend. I've been to all of them so far, and I'm happy to be going to this one thanks to the generosity of my evil twin Lisa (we're not really twins, but people kept asking if we were, so we ran with it). Anyway, to make a short story long, I do two main activities at comic book conventions. I ask nicely for free sketches, and have gotten a few, and I troll people. Both activities result in physical proof, one in the form of a sketch and the other in the form of a photograph. But until this year I've never been able to bring all the proof along with me. Two of my sketchbooks are completely filled and one of them is utterly priceless to me because of the sketches in it. I leave it in my firesafe unless I want to look at it. Bringing it to every convention so I can show off the awesome sketches in it isn't an option. I have scans of all the sketches but I haven't got permission to share them all online, so most people will never see some of the artwork in my sketchbook. As for the Torvald pictures, most of them are available online but I don't have easy access to the internet while at shows. My highest tech portable object is my Kindle. So again, until now, I had to be content with directing people to the website.

A few days ago, though, I suddenly realized that with my Kindle and Calibre, I could create ebooks for both the sketches and the troll, and be able to show them off freely, if in plain B&W, to anyone who wanted to see a particular item. It might be a little tricky to make sure I could navigate to the particular picture I wanted, but I have already played around with the ePub format a little, so I thought I'd see if I could do it.

The ePub format, fast becoming a standard for ebooks, is a very simple one. It's basically a set of html files zipped up. That's it. Now, I've been making webpages for over 15 years, so I have a pretty good grasp of basic HTML. Even better, the Torvald site was already set up in such a way that I figured it would be a piece of cake to convert it to an ePub. So I started with that. I removed all the extra formatting from the page, and simplified the layout a little. Then I zipped up the site and imported the zip file into Calibre. Once there, I converted the zip to ePub and took a look at the result.

It was a shambles. The links didn't work, and the layout was completely different. Oops.

I took the ePub that Calibre had created and examined it by saving it to my disk (right click, save to disk) and then changing the extension to zip from epub. It had a handful of extra files in it, and it had removed all my extra picture folders and put everything in the main folder. I went to another epub I'd already dissected and looked at it to see what I was doing differently. It took a little sleuthing, but I discovered that my internal links were using a different convention than the ones used by the working epub. I'd been taught to make a link to a specific place inside a page by making an "a name" tag and putting it around that place. The working epub's HTML files instead used an "id" tag within another tag. So I changed all of Torvald's internal links to id tags instead of name tags, removed a bit more garbage HTML code, and tried again.

This time it worked. I had an epub book with a table of contents that I could click on a name, and be taken to the picture of that person with Torvald. It was awesome. I converted it to mobi format and uploaded it to my Kindle, and it still worked. It's a HUGE file, 25 megs, but it's there on my Kindle so if anyone at Emerald City wants to see some particular person with Torvald, I can click a couple of times and bring up that image.

The sketches were harder. The website isn't one that can simply be converted, and as I mentioned earlier, all the sketches aren't on the website anyway. Plus, with these, I wanted them to be as big as possible. So I figured out what the maximum size of an image within a book is on the Kindle (520x640), found my raw images, and did a big of cropping, editing to make the contrast better, and some batch resizing and converting. With the images, all 278 of them, ready, I had to create an HTML frame to hang them on.

I divided the collection into eight sketchbooks (three Aquaman ones, an autograph one, one for images not in sketchbooks, a Doctor Who sketchbook, my husband's sketchbook, and my evil twin's sketchbook). Each sketchbook got its own page, and I made the HTML as simple as possible. Just a embedded pictures with id tags so the index could find them. I opened each page in Firefox as I finished to make sure the page were displaying properly.

Then I made two indexes. The first one pointed to each of the sketchbook pages and also to an A to Z index of the artists, which was the second index. I set that one up alphabetically, linking each artist name to the picture by the id tag. It was a bit time-consuming work, but it was to pay off.

Once I was done, I zipped it up and imported the zip file into Calibre, then converted it to ePub. The Table of Contents was a mess. It tried to include part of the A to Z index instead of just the main index.

So I dissected the ePub and copied some of its files over to my original work. There's a toc.ncx file. I edited that one down to just the items I wanted to show in the index. There's a content.opf file. That one just lists all the files in the epub, so I mostly left it alone. I also copied over a META-INF directory and it's sole file as it was, and a mimetype file as it was. Calibre also created a titlepage.xhtml file, and I copied that one over as well, because it's the file that creates the book cover. With the edited toc.ncx file, I zipped up the contents again. This time I renamed the zip file to epub, and imported it into Calibre as an epub file. It worked. Perfectly.

Next job, convert to mobi and upload to my Kindle. And it all worked. I can find an artist name in the A to Z and jump quickly to it.

So I created a couple of books for my Kindle. One from a website, and one from scratch. I'm feeling pretty good about it at the moment, though I'm far from knowing everything. I admit, I can hardly wait to get to Seattle and see my friends and family, and enjoy the convention.

If you happen to be going and want to see my efforts in person, just look for the gal in the Aquaman t-shirt carrying Torvald the Troll in a belt-pack.

Kindle and Calibre - DRM and Plugins

This is very much the elephant in the room of ebooks. I make no secret of the fact that I despise Digital Rights Management (DRM). It's an attempt to use technology to solve a social problem. As a result it is always quickly broken and instead of stopping the copyright infringers it makes it hard for honest people to use files they have already purchased. I've already posted about a situation where I attempted to remove DRM for someone, but failed and ended up pirating it (to my personal dismay). There has been at least one study that shows people will turn to piracy to get what they need out of ebooks that have DRM. The online comic XKCD has a persuasive argument why DRM makes criminals out of us all. So I'm not fond of DRM*.

In short, DRM only stops honest people from being able to use a file the way they expect to be able to. It does nothing at all to stop the "pirates". The people who want to make a file widely available will always be able to break the technology used to create DRM, and as a result the pirated copy of a book will always be more useful than the one laden with crippling DRM. As a publisher, you NEVER want a pirate to offer a better product than your own. That's stupidity in marketing.

As an aside, I have repeatedly argued for a different kind of DRM scheme that embeds a user's information in a file in multiple ways without restricting the file's use (watermarking). I have absolutely no issue with a file I download having information that will allow a company to see if I have put that file on a file-sharing site. I don't share my files anyway. Such a scheme would not cripple the file, meaning that normal non-techie people can use it fine. Yes, it's still technology and therefore people could still remove the ID from the file and pirate it, but it would be less likely to happen, in my opinion. People would rather be honest than dishonest.

Getting back to the discussion at hand, while some publishers offer DRM-free books (such as Baen Books), most cripple their books, making them impossible to read on multiple devices unless you register every device and download them specifically to those devices. For many bookstores, you can't register a competing device, so if you have a Sony and want to read a book you bought for Kindle on it, you're out of luck.

Let me state this right now: Calibre will NOT help you break DRM. Calibre is a management program, not a piracy program.

However, Calibre is an open source program that allows plugins, and there have been multiple plugins created for Calibre that will remove almost any type of DRM from any file you import into Calibre. I want to stress: this is not legal everywhere, and any justification you make for it is an excuse. If you decide to hunt down anti-DRM plugins for Calibre, do so fully aware that what you are doing is probably illegal and possibly immoral as well. You may disagree with me, but I won't lie to myself. What I did to help Tim and Jane was wrong, even if it was moral in some sense. And if I were to, say, remove all DRM from every book I purchase it would also be technically illegal, even though I do NOT share my book files.

If you really want to be a bad consumer and strip DRM from your ebooks you will have to use an internet search to find the files you need, because I cannot and will not link to them or host them here. First, Amazon has an understandable desire to sue people who help others to break DRM. Second, DRM tech is constantly advancing as the DRM peddlers continue their futile attempts to stay ahead of the techies who enjoy removing stupid things like DRM. So a link to a DRM removal tool would soon become obsolete anyway.

Side Thought: DRM peddlers are like drug dealers. "Sure, c'mon, we can get rid of your nasty hallucipirates, just give us a lot of money and we'll give you our latest and greatest." And the DRM might work for a bit, but they will soon have to return for another hit.

If you are determined to strip DRM from your Kindle ebooks, here's some hints. First, you may want to install Kindle for PC or Kindle for Mac. And you might want to download any ebooks you want to strip of DRM to those programs. When you do an internet search, you maybe should look up something about Calibre, plugins and DRM. You should not have to sign up for any forum or pay any money for the plugins. I can't tell you what exact searches will work, or if you'll be able to find what you are looking for. But hey, if you are determined, good luck.

One last note: If I learn that you've used this information to pirate ebooks and share them with everybody, I'll consider you lower than pond scum and scorn you for the rest of your life. If you want to remove DRM, remove it for private use, not copyright infringement.

Let's move on to a less dangerous subject. How about, oh, Calibre plugins!

Adding a plugin to Calibre is very easy. First, you find the plugin you want to add. There's a good list of plugins here, and they include some really neat stuff. For Kindle 3 users, check out the collections management plugin.

Each plugin comes in the form of a zip file (don't extract it, it needs to stay a zip file). Simply download it to your computer somewhere you can find it again.

Next, you need to tell Calibre to load up the plugin.

• Click on the Preferences button.
• Then, down in the lower left should be a "Plugins" button. Click that.
• At the bottom of the window that opens is an "Add a new plugin" button. Click it, and you'll get a search window.
• Find your downloaded zip file, click "Open", and let it install. You might get a warning about the danger of a plugin. As long as you downloaded the plugin from a trusted source, you should be ok and can click through.
• Now your plugin will show up in the list of plugins. At this point, if it needs to be customized, you can select the plugin and click the "Customize plugin" button. What you get will depend on what features the plugin has, if any. Many plugins don't require any customization.
• Once you'll installed and customized, if needed, then you need to restart Calibre. You can do that quickly by typing Ctrl+R.
• It is safe to delete the original zip file you downloaded once the plugin has been installed.
• If the plugin needs to have a menu item or toolbar button added, go to Preferences and then "Customize the Toolbar". Use the dropdown to find what menu or toolbar it needs to be added to, then find the action in the "Available actions" box, select it, and add it to the other box. Use the arrows to move it to where you want it to show up.

Once you've installed a plugin and restarted Calibre, the plugin should be working immediately. A handful of plugins, like the collections one, require a couple more steps to make them work best. Read the instructions that come with the plugin to make sure. Quite a few plugins don't require any effort on your part at all, they just work.

If you are a developer yourself and familiar with Python, there is good information about how to create a Calibre plugin in the Calibre User Manual.

That it for these topics. If I ever get a Kindle 3, maybe I'll write up how to actually use the collections plugin to your best advantage. But my poor little K1 doesn't have collections on it, so I'm clueless about it at the moment.

* There is one single exception to my distaste of DRM, and that's library books. I may go into detail later, as I intend to discuss library books and the Kindle, but at the moment it's enough to say that I don't mind DRM on library books.

Kindle and Calibre - User Manual

The Calibre User Manual is available as an ePub download. Once you've downloaded, you can convert it to mobi format and upload it to your Kindle to read at your leisure.

Kindle and Calibre - Free Books

Everyone likes a freebie, and with the Kindle I can actually read freebies without eyestrain. And so I'm going to list a few of my sources of free ebooks, so you have an idea of where to go to get free books if you want to join in. The Calibre side of this is pretty simple, actually. Any ebook that isn't in Kindle format and doesn't have DRM (Digital Rights Management) on it can be converted to Kindle format with Calibre. I've actually learned how to create and edit ePub files myself, then convert them to mobi format to upload to the Kindle. But more on that later.

Amazon.com itself has a bunch of free books, they can be found here. They have two distinct collections of free books. First is the classics, which are in the public domain and always available, and the other is a collection of limited time offers. The limited time offers change on a nearly daily basis, and some of them are only available for a few hours, so it's best to snag those the instant you see them.

If you don't have a Kindle, there are a number of different Kindle applications that will allow you to read Kindle books on your favorite device. Before I got a Kindle, I used Kindle for PC to download a handful of free books, but I couldn't read them very well because of the eyestrain problem. When I registered my Kindle the first time, those books were automatically uploaded to my Kindle, giving me a nice starting place to read.

Of course, Amazon is hardly the only source of free e-books online. The biggest and bestest would be Gutenberg, a repository for out-of-copyright works saved in digital formats that are easy to work with. A lot of those books already have Kindle editions ready, and they can be downloaded and added to Calibre with ease. Best of all, no DRM so you don't have to worry about whether or not the book will open on any new device you get.

One of my favorite sites to get free e-books is MobileRead.com. The care that has gone into making these out-of-copyright books into nicely formatted reads often shows quite plainly. I love getting huge omnibus editions of a writer's works, for instance this Oz Omnibus. I've found that the best way to get the most from MobileRead is to look for the HTML link over on the upper right, where it says "Full List", and open that file in a browser window. Then I can find neat stuff by scanning down the list.

I would be amiss if I didn't mention the Baen Free Library. This was started many years ago by Jim Baen (bless and rest his soul) who believed that most people would rather be honest than dishonest, but DRM was so restrictive that people would be driven to pirated material in disgust if they couldn't get what they wanted from an official source. And he strove to make sure that every e-book he published was exactly what a customer would want. Then he went a step further and started to give away ebooks, usually the first book or two of an ongoing series. That was the Free Library, and it's continued ever since. And if that weren't enough, then Baen started giving out CDs with hardcover books, CDs that readers were encouraged to share that often included pretty much an entire series of books. Yes, it's legal. Yes, Baen knows about it. The books are provided in a variety of formats, making it easy to read them on just about any device you can imagine.

Frankly, between just those sites there's enough material to keep me reading for many more months, but there are a handful of other sites that are useful as well. ManyBooks has a few nice free books, although a lot of its archive is from Gutenberg originally. Smashwords has a lot of original books for free. The site is a publishing platform for independent publishers and authors, so you may find all kinds of interesting stuff there.

If you want a one-stop sort of site, Inkmesh might be your place to go. The site is an ebook search engine, and includes easy links for a number of free ebook sites.

A couple of sites that will help you monitor free books are Daily Cheap Reads and eReaderIQ.

And that's just the ones I can think of right now. Many authors provide either free books or samples on their personal websites. Amazon.com also offers samples of almost all Kindle books, usually the first 10% of the book.

Update: A new site called Open Books offers a few books that are free, but more importantly, all the books linked from the site are free from DRM, so they can be read on any reader.

Update: NetGalley offers books for free, but there are a few caveats. The first is that these are review copies, not always fully edited. The next is that the publisher really wants you to write a review and post it somewhere. The last is that there is no guarantee that the publisher will choose to give you a review copy. I've gotten a couple of books through them so far, and both have been excellent in their own ways.

So, that's my free ebook sites. Do you know of any other sites that offer free ebooks on a regular basis that you might want to add to my list?

As this is an archive for my own use as much as my readers, here's a list of links to the sites I find useful that I might update if anyone adds any more useful sites in the comments.

• Kindle Free Books Collections
• Gutenberg
• MobileRead.com
• Baen Free Library
• Baen CDs at the Fifth Imperium
• Smashwords
• ManyBooks
• Inkmesh: ebook search engine
• Daily Cheap Reads - ebook tracker
• eReaderIQ - ebook tracker
• Open Books - not all free, but all free of DRM
• NetGalley - Review copies for reviewers

Kindle and Calibre

I installed Calibre back when it was recommended by Wil Wheaton on his blog, and I found I didn't use it at all. The file structure was strange, and I only had a few ebooks to put into it. I didn't realize it had a viewer, and in short the usefulness of it seemed extremely limited.

But now I have a Kindle, and I suddenly understand why Calibre is so well-loved by the literati of the ebook crowd. Yes, the file structure is odd, but that's because the program is looking at the files as books, not as files. And it can be hard to get it to work if you aren't technically competent, as the manual isn't written for the average user. That's due to it being written by a computer engineer for computer techies. But more people are going to be using it in the future, so I thought I'd start to work on writing up some of the tricks of using Calibre so I can remember what I did, and other people might be able to benefit from my experiences as well.

Today's first trick: downloading and installing Calibre.

Ok, not much of a trick. If you've done any installation at all, you know already what to do. The Calibre downloads are available for lots of different operating systems, and I've installed it on both Windows XP and Ubuntu myself. At the moment, I'm mostly using my Windows install, so my tricks will be written with that in mind.

Now, when you install it, Calibre will create a library, and THAT is a very key point. I have several libraries for my books. I have a main library that is actually located in my Dropbox and holds only the stuff I'm currently working with. On my external drive I've put several libraries that have my archives, my backups, and some other stuff. I'll get into those more in the future. But the location of the library is important, if only because later in my tricks I'm planning on explaining how I made a bunch of different libraries and why.

In any case, once you've installed Calibre, you'll want to know what to do with it. Well, the best thing is to watch the video tutorials, as those will give you a good start on figuring out the program, and introduce you to Kovid, who is the guy who built this fantastic system and is giving it away for free (though donations are accepted). The tutorials use an older version of Calibre, but still manage to give a good idea of how it works.

I'm tempted to leave it at that, because if you watch the tutorials you will probably be able to figure out the rest without difficulty, but I know that most people won't bother with the tutorials. So here's some of the basics: Use the "Add books" button to add books to Calibre. It copies the books from whatever location you get them from into it's own file structure. The original is left alone. Once in Calibre, you can edit just about everything that has to do with the file by clicking the "Edit metadata" button, or right-clicking on a book title and selecting "Edit metadata".

The metadata is actually where the magic of Calibre sorting comes in, as you can sort on just about every field in the metadata, and you can add your own custom fields and sort on those as well. For instance, I like knowing where I got an ebook from, so I added a field called "Source" in which I put where I bought it, or if I got it from Gutenberg or another free site. Adding a custom field is simple, you go to "Preferences", then "Add your own columns". The lookup name must be one word and lowercase, otherwise you can play with the settings and come up with what you want. Other fields I added include a boolean (yes/no) column for whether or not I've read the book, whether or not I've reviewed it, and in my Dropbox library I have a column to indicate if I've backed it up to my external drive.

Well, I'm going to write more on this topic, but at the moment I think that's enough to get someone started on the program. If you have any favorite Calibre tricks you'd like to share with me, feel free to post about them in the comments. I'll be covering Calibre news and blog recipes, how to get a series title to show up as part of the title when you upload to Kindle, and using multiple libraries while still having a nightly news download in future installments.