I use Kovid Goyal's program "Calibre" to manage my eBook collection, which currently runs to more than 4,500 books.

Just in the past few weeks I have encountered a problem with the program which I believe is NOT a bug in Calibre, but something going on with my computer.

Under normal circumstances, it takes between four and five seconds for Calibre to load and display, ready for input, after I click on the Calibre icon. However, if it is the first time loading the program following a reboot, I get a 50-55 second delay while the program says it is "Initializing user interface...". This only happens following a reboot or restart of the computer.

Kovid Goyal responded as follows:

"This will be because of caching/preloading. Windows maintains a cache of the DLLs/data files used by recently/frequently started programs in RAM, allowing them to start up again quickly. Presumably something in your windows installation changed and that cache is not being prepopulated after a restart with the DLLs for frequently used programs."

I am running a recently (~three months) clean install of Windows 10, and Calibre ran fine in that installation until quite recently.

This kind of thing is way beyond my pay grade. What do I do now?

tanstaafl.

The caching thing is a red herring. Dude thinks you've got some kind of old/slow computer. You don't. No program should take 55 seconds to load on a fairly modern system.

The only way the caching thing that he describes would be true, is if somehow your disk drives were super slow. Slower than tar, slower than they should be on any modern system. Like... A SATA drive that is running in compatibility mode because it doesn't have a device driver loaded or it wasn't detected by the BIOS correctly or something. If that were the case, you'd notice performance problems across the board, not just in that one application. (do you?)

The only times I have ever had a program do the "slow startup" thing, it was because that program had been used to open a file on the network or on a removable drive somewhere in the past, and now that old file was in the program's list of "recently opened files", but now that network share or that removable drive is gone, so now the program tries to start up but it sits and hangs and waits for 30 seconds on each instance because it is timing out on the network or disk query for that (now gone) location.

Try: Clean out the list of recently-opened-files in that program.

Or, look for some other thing that the program might want to be opening (some kind of file or folder or library or network printer or web site or something) that is now gone/deleted/dead/disconnected/slow, that might be causing it.

One other question: Is the "Calibre" program, when it starts up, loading up your main ebook collection by default?

If it is, can you make it so that it doesn't do that, so that you can separate out the two possible steps?

- Clean loading of Calibre
- Loading up your particular eBook file

If you can get it to just do the first thing, if that thing is what's taking 55 seconds... yeah the caching thing is a red herring.

If it's the loading of the eBook file that takes 55 seconds, then that's a different question. There could be something weird in the file, like a circular database reference or something.

This is a reasonably fast computer, 4.10 GHz 64-bit dual core processor, 8GB RAM, system drive is an SSD, SATA hard drives, clean recent Windows 10 full installation (not upgrade). There is no performance degradation anywhere other than Calibre startup after reboot. The computer will go from power-off to reading my email in 22 seconds.

Yes. That is what the program is for -- to manage the collection. It is not loading the ebooks themselves, only the books' metadata which points to the actual books on a different hard drive.

Yes.

Here's what happened.

1) I renamed the directory where the actual eBook files are kept.
2) I did a power-off reboot.
3) I started Calibre.
4) Instead of the usual 3-5 second load time (when not loading after a fresh reboot) it loaded in two and a half seconds. I closed the program.
5) I renamed the directory back to its normal name that Calibre is configured to look for.
6) I started Calibre again. It took 25 seconds to load, instead of the usual 3-5 seconds. This was NOT following a fresh re-boot.
7) I closed Calibre and re-started it. It took the normal 3-5 seconds to load.

I am unclear about what you mean when you say "Loading up your particular eBook file." Calibre is not a program for reading or editing eBooks (it can do those things but I don't use it for that). It is primarily for storing metadata on each book and connecting to my Kindle and transferring files to and from the eReader, and editing the metadata. It is useful for converting eBook file formats, say from epub to mobi. But none of that can be done until the program finishes loading, normally 3-5 seconds, or in my case 55 seconds after a re-boot.

Is this information helpful?

tanstaafl.

Here is another response I got from someone who saw my original bug post on the Calibre website. Again it's above my pay grade, but maybe it will make sense to you.

Charles Haley 1650803@bugs.launchpad.net via canonical.com

to me
Take a look at this thread on mobileread. The link starts you at post 130,
where the relevant info begins.

http://www.mobileread.com/forums/showthread.php?p=3119974#post3119974


Is there anything useful there?

tanstaafl.

By "ebook file" I meant whatever the kind of file that Calibre loads up.

So you have narrowed it down to a problem loading your file, not a problem loading Calibre itself, right? Calibre loads quickly when it's not loading your file?

You have also narrowed it down to a slowness bug in Calibre, not a slowness bug in your general operating system, right? Since Calibre loads quckly, it's only loading your file that's the problem, and other programs don't have this issue?

There's something Calibre is doing that is being slow, and it's doing it when it loads up your file.

That thread you linked talked about BONJOUR a lot. Bonjour is an APPLE program that would have self-installed at a time that you installed some other related Apple product such as iTunes. I have, in the past, found it to be buggy, and I have had slowness problems with Bonjour. It is a networking program which inserts itself in your network stack, and it has sometimes had problems on certain kinds of networks which slows down my computer when it makes certain kinds of network calls. For instance, I had a work PC which, when it was joined to the work domain, would take 60+ seconds to shut down because it was trying to synch my profile to the domain at shutdown time, and Bonjour would get in its way because Bonjour was trying to shutdown at the same time and it was blocking the network stack. My solution was to uninstall Bonjour. Bingo, quick shutdowns again.

Run APPWIZ.CPL, see if BONJOUR is installed, and remove it.

If that solves your problem, then you have indeed had the same problem as the people in the thread that you linked. The problem is that somehow Calibre is doing something on the network and it's trying to either talk TO bonjour or trying to talk VIA bonjour and the buggy-ass not-properly-QA'd bonjour program is getting in the way and causing the slowness.

Please note that Bonjour likes to reinstall itself each time you upgrade an apple product. Lately I've been letting it live on my system because I keep upgrading iTunes (I need iTunes to make backups of my iPhone), but any time I have something that looks like a network slowness problem, the first thing I do is uninstall Bonjour and see if that fixes it, because that's been my experience in the past.

A different person on the thread said that their problem was resolved when removing Adobe Font Manager (atmfd.dll) on their system.

I've also, over the years, had troubles with Adobe products getting in the way of system processes much like the troubles I described with Bonjour. It wouldn't surprise me if that was your problem as well, if you'd ever installed any Adobe products on your system which might use that.

It would appear so. When I rename the directory that contains all the actual eBook files Calibre starts up normally, whether or not the computer has been rebooted since the last use of the program. In the process of starting up, it creates a new, empty eBook directory.

Does this suggest that perhaps a file in the eBook directory is corrupted and causing the delay? The directory tree is: D:\Library\eBooks\[Author name]\[Book Title]\cover.jpg, and [title.mobi] and metadata.opf. The title.mobi file is the actual eBook.

Since this eBook library is four or five yeas old (or older?) and is added to weekly, if there is a corrupted file I should be able to find it by sorting by date modified and moving newer directories out of Calibre's search path until Calibre works normally again. Remember, everything worked right until just a few weeks ago, nothing changed except the addition of new eBooks to the library.

I do not allow iTunes anywhere near my computer, thanks to your pointing me towards "CopyTrans Manager" several years ago. (Couldn't you use CopyTrans to back up your iPhone?) I ran the APPWIZ.CPl, and there is no Bonjour. I also did a search for "Bonjour" on my C: drive just to be sure, and the program is definitely not there.

I have Adobe Acrobat 9, plus Adobe Reader in my computer. They have been there for years, carried over from my Vista system into my 8.1 system and now into my Windows 10 system, and they have co-existed with Calibre during all that time. There is no atmfd.dll file anywhere on my hard drive.

Since, as Mr. Goyal said, something must have changed in my system to cause the problem, I can only think of two things that are different: A Windows 10 update, or the addition of a new, corrupted eBook file. The second option seems the most likely.

Or am I looking entirely in the wrong direction? The fact that the program loads normally with an empty eBooks directory, but loads slowly when the directory is populated, is suggestive.

tanstaafl.

Can you split the library in half and see if one half is good and one is bad?

If so, then split the bad half and add to the good and see what happens. Keep splitting the bad half. With 4000 odd books it would narrow it down within about 12 goes.

I'm not sure a corrupt book file could cause such big delays though.

Could it be reaching out to a non-existent URL that is timing out?

Funny y9u should ask. I just finished doing that, but rather than split in half I just moved out of the Library all new acquisitions since October because it only started failing sometime in November.

It didn't take long to narrow it down to a single file, and it was not one of the eBook directories or files. It was the metadata.db file.

While Calibre keeps the meetadata in a separate file for each eBook within the directory for that eBook, this information is just backup for the metadata.db master file which is kept in root of the Library directory. The metadata.db file is what is loaded when Calibre starts.

If I start Calibre without the metadata.db file in place, Calibre creates a new, empty metadata.db file and the program loads and starts up very quickly, but of course without showing anything from my eBooks library.

From there, there is an option within Calibre to rebuild/restore the database by going to the metadata files associated with each eBook amd recreating the master metadata.db file.

I have done this. The process takes a couple of hours and failed the first two times I tried it because it aborted when my power-save settings put the computer into sleep mode part way through. Ooops!

The new metadata.db file is now created, and it is about half the size of the old one. Calibre now starts up and runs in the normal three seconds... UNLESS the computer was rebooted since the last time the program was run. Then it delays the startup for about half the time it used to delay, no doubt because the new metadata.db is about half the size of the old one.

There is a ray of hope, albeit a small one. The restore flagged seven of the 4500 books (OK, I exaggerated, there are only 4489 eBooks in the library) as having metadata errors. I can delete these books from the library and see if that changes anything. I am not optimistic. I have the original source material for three of the seven books, and only one of the remaining four books (Larry Niven's "Footfall") is important enough to me to want to re-source.

Time to reboot and see what Calibre does with those seven books now out of the library.

======================================

Edit: Well, that was a surprise. Immediately after a reboot, Calibre loaded and ran in two and a half seconds once the seven books were removed from the library.

I think I can say "Problem solved."

tanstaafl.

If you can narrow down what exactly caused those 7 ebooks to break Calibre, you could submit that to the author as a bug report. Future versions of Calibre could detect the presence of the problem (whatever it is) when indexing the books, and correctly compensate for it, preventing others from running into the same slowness issue as you did.

I did that last night.

tanstaafl.

FYI Calibre is basically a two part system. A web server and a file/metadata engine. The don't even have to run on the same cpu as long as each half is pointed at the same directory.

So what specifically was wrong with those 7 ebook files to cause the issue? I'm curious.

The eBook files were fine. The metadata for the eBooks that was kept in the master metadata.db file was corrupt. The metadata for the individual eBooks, kept as part of each eBook directory, was not corrupt, so Calibre was able to rebuild the master metadata.db file from the eBooks themselves. Although I deleted the seven books from Calibre before rebuilding the database, that actually was not necessary. I was able to add all seven eBooks back into Calibre after the rebuild.

The corruption in all seven books was the same: even though the eBooks were physically present in the library with clean metadata, the master metadata.db file said there was no valid eBook format available for that particular eBook.

tanstaafl.

Interesting. So, whatever it was, it's not a repeatable bug, I gather? Something went wrong when adding those particular 7 books to the database, but adding them a second time didn't reproduce the problem? Too bad. Intermittent bugs are the hardest to locate and fix.

Ummm.. not exactly. Those seven books had been in the library for years, but the delay problem was only weeks old, and started suddenly, that is, one day everything was fine, the next morning the 55 second delay.

The only commonality I can find is that each of those seven books had been modified after being added to the library -- converted from .epub or .mobi into .rtf format so I could edit the books, then the edited file converted back into .mobi for the Kindle. But there are dozens, no, more likely hundreds of other books in the library that have gone through that process.

tanstaafl.