Hey, I thought I would start a new thread for this since the other one I had going didn't have a very helpful title. For those that missed it I managed to get ext3 working on the empeg. Ext3 is a journaling filesystem which has the advantage of not needing to do a long annoying file system check if the empeg crashes during a sync or you forget to mount the drives read only after modifying something. This thread is meant to be a writeup of how to do it in case anybody else is interested. First a few warnings though...

1. This is a very old version of ext3 and there are bugs in it
2. I had to upgrade the empeg kernel to get this to work. Many of the patches did not apply cleanly so I had to hack in any code that didn't get patched by hand. It is possible I made mistakes.
3. I don't know if this version of the linux kernel will cause stability or other problems with the empeg hardware.
4. If something goes wrong this could eat all the files on your empeg
5. This will break IR support (I am hoping somebody here will figure out why)
6. I take no responsibility for any bad things that happen because of this, if you don't understand the directions don't try this.

First of all the bits and pieces. All these files are gzipped so you will need to uncompress them before using them...

1. I had to upgrade the linux kernel from version 2.2.14 to version 2.2.17. A patch to upgrade the empeg kernel tree to that version is at http://macgeek.dyndns.org/empeg/ext3/empeg_v211-linux-2.2.17.patch.gz
2. Next patch is for ext3 support, a patch for that is at http://macgeek.dyndns.org/empeg/ext3/ext3-0.0.3b.patch.gz
3. I also tweaked hijack a little bit to patch on top of the above. A patch for that is at http://macgeek.dyndns.org/empeg/ext3/hijack281+voladj+rds.patch.gz This is optional.
4. A kernel built with all of the above for a mark2 is at http://macgeek.dyndns.org/empeg/ext3/zImage-2.2.17-empeg-ext3-hijack.gz
5. I don't know if the version of fsck that ships with the empeg will be happy with ext3. A newer version of e2fsck that understands ext3 is at http://macgeek.dyndns.org/empeg/ext3/e2fsck.gz You may not actually need this, but it can't hurt.

Ok, so to get this to work first install the kernel and then follow the steps below.

1. Put the new version of fsck on your empeg. I put mine in /bin. Don't forget to make it executable. You should also make the other fsck files links to this to make sure the right ones get called like this...
empeg:/# mv /bin/fsck /bin/fsck.bak
empeg:/# mv /bin/fsck.ext2 /bin/fsck.ext2.bak
empeg:/# ln -s /bin/e2fsck /bin/fsck
empeg:/# ln -s /bin/e2fsck /bin/fsck.ext2

2. You nead to create a journal file on the target drive. There is a utility called tunefs that is supposed to do this, but of course it did not work for me. Here is the hard way to do it...

# First turn on swap, I got a panic using dd without it
empeg:/# swapon /swapfile
Adding Swap: 16596k swap-space (priority -1)

# Make the drive read-write
empeg:/# rwm

# Use dd to make a file for the journal, this creates a 10meg journal file.
# There may a more optimum size
empeg:/# dd if=/dev/zero of=/drive0/.journal bs=1k count=10000
10000+0 records in
10000+0 records out

# Figure out the inode number of the new file. It is that first number.
empeg:/# ls -i /drive0/.journal
12 /drive0/.journal

# Unmount the drive
empeg:/# umount /drive0

# Remount the drive as ext3. The first time you mount the drive
# you need to specify the inode number from above with the journal
# option.
empeg:/# mount -t ext3 /dev/hda4 /drive0 -o journal=12
EXT3-fs: recovery complete.

3. You probably are going to want to update your /etc/fstab file to mount the drives as ext3 instead of ext2.

4. This is the tricky part, we need a way to get the player to use ext3 instead of ext2 when it remounts the drives during a sync.

The first way to do that is to edit the player executable and change the string that says ext2 to ext3. For me this causes a stack trace when the player remounts the drive. It sounds like a good idea, but doesn't work at least for me.

The second way would be to write a kernel patch to intercept the mount call from the player and fix it to do what we want. I am hoping Mark Lord can help out here since he was talking about how to do this in another thread. Mark, you around these days?

The third way would be for the empeg guys to help us out and make the player a little smarter about how it mounts the drives.

5. You may need to replace the empegs init with a script that mounts the drives as ext3 instead of ext2. I already was using a custom script so I didn't worry about this.

Let me know if you have questions or suggestions on the IR driver problem. Those files are only on a DSL line so downloads will be kind of slow.

-Mike

One correction. IR does work fine with this. One of the remotes I was trying to use must have had a dead battery and the other required hijack which I hadn't patched in yet. D'oh! I was pleasantly surprised when I got in the car this morning and my normal RIO remote worked just fine.

-Mike

OK, I managed to figure out how to change the mount type within the kernel. This intercepts any calls to mount hda4 or hdc4 and changes the mount type from ext2 to ext3. The patch is at http://macgeek.dyndns.org/empeg/ext3/mountHack.patch.gz and a new kernel with this change is at http://macgeek.dyndns.org/empeg/ext3/zImage-2.2.17-empeg-ext3-hijack.gz

So... that should be everything needed for anybody interested to get ext3 working on their player. I have been running it for a couple of days now and have done at least a dozen syncs with no problems. It rocks!

-Mike

p.s. to mlord. You can ignore my private message, as you can see I managed to figure it out.

So is there really nobody interested in this? My server shows 0 downloads after 3 days. It seems like there where a few people interested in the other thread I had going. Everybody likes 30-90 minute drive checks? Weird.

Honestly, the only times I've ever had to fsck were times that I was logged into the empeg and forgot to ro it before rebooting. That's my own damn fault and I should be penalized.

I meant to ask earlier what sort of ext3-specific options the filesystems were being mounted with. (Not that I'm sure that it makes a load of difference in this case, since it's just the journalling type.)

Oh, I'm very interested. I just haven't had time to really digest the implications of installing it wrt to any hijack work I want to do. I don't have the luxury of a spare for testing

I'm also not convinced that you needed to change the kernel to trick the player into mounting the ext3 parition during sync. Doesn't the player use /bin/rwm and /bin/rom to achieve this? Couldn't you have edited etc/fstab to also achieve this? I'm asking because this trick is pretty absolute. It means always running a custom kernel to use ext3, and always running ext3 if you are using this kernel, and I was wondering if we could persuade Mark to consider including your work in hijack.

The fstab file is an unused stub, so it won't do anything useful. Not sure about rwm and rom.

However, ext3 is a superset of ext2 and you can switch which way you mount the filesystem back and forth between them with no ill effects at all.

I meant to ask earlier what sort of ext3-specific options the filesystems were being mounted with.

None. Honestly I never bothered to see what other options I may have. I am passing through whatever the empeg guys are using except for changing the type to ext3. Is there anything else useful that I should investigate?

-Mike

I'm also not convinced that you needed to change the kernel to trick the player into mounting the ext3 parition during sync. Doesn't the player use /bin/rwm and /bin/rom to achieve this? Couldn't you have edited etc/fstab to also achieve this? I'm asking because this trick is pretty absolute. It means always running a custom kernel to use ext3, and always running ext3 if you are using this kernel, and I was wondering if we could persuade Mark to consider including your work in hijack.

I couldn't come up with any other way to modify the mount type. I even tried using a custom /bin/mount, but that isn't getting used either. The player does not use the rom/rwm scripts, it calls the kernel function directly and never even looks at fstab. As someone else mentioned it is trivial to move back and forth between ext2 and ext3 so that shouldn't be an issue. I was hoping Mark would step up and make the ext3 mount configurable within hijack so that it can be turned on and off (kernel programming isn't really my thing). I was also hoping he would consider including all this within hijack so I don't have to maintain it separately. But, there are a lot of changes there so it could definitely use some testing first and I am currently the only person using it so that is why I was hoping to get some more interest going here.

-Mike

So is there really nobody interested in this?

I am, but I have been way too busy to even get a serial cable near my empeg. I still need to get d.net working for PaulWay, and some other things done.

Here, you take over the BBS/Riocar.org for a month or two, plus work one of my days at work and I might get to it soon :-)

Seriously though, glad to see this up and running after my failed attempt at ReiserFS a while back. Once I do start hacking the empeg again, ext3 mounts after a crash will be a welcome sight.

You do need to change the kernel. The player uses the mount syscall.

I'm kinda slow tuning in here. Nice job, by the way.

The part that interests me the most is the 2.2.17 kernel upgrade. I'm always interested in newer kernels for some strange reason.

The ext3 is a nice idea, and could be very worthwhile for many of us (PaulG ?). Personally, I'll pass for the moment (just too busy with other $tuff), but if I did convert everything over to ext3, then I'd also nuke ext2 completely from the kernel if possible, to save RAM.

The only glitch I can think of with that is the software upgrades (future) will expect ext2 rather than ext3, and may choke on the superblock since they use an unmodified kernel.

Cheers

I agree, this is very cool. Since I got my Mac, however... I have no serial port, and I don't have time to check this out. The reason that this mainly is cool, as opposed to XFS, reiser, or JFS, is that nothing has to be done to convert the fs (well, other than tunefs, I mean you don't have to reformat) and that an ext2 kernel can mount it if need be.

...but you can't tunafish.

--
The Ghost of man-pages Past.

Ok, I downloaded it today and was just now about to enter the installation process when I saw a new beta of the player ... So being that I want to cause the most hassle and most flakiness, what's the process by which I put ext3 on the new player software?

Greg

Ok, I downloaded it today and was just now about to enter the installation process when I saw a new beta of the player ... So being that I want to cause the most hassle and most flakiness, what's the process by which I put ext3 on the new player software?

:-) Keeping in mind that I haven't actually tried b13 yet, the install process shouldn't have changed from what I posted in the original couple of messages in this thread. You might want to wait until I get a new kernel built (probably in the next hour or so), but it doesn't sound like there were many kernel changes in b13.

-Mike

OK, latest and greatest files are now available for 2b13 and hijack 283. MountHack, ext3, and (obviously) e2fsck did not change. The rest didn't change much. I probably won't be able to test the new version until tomorrow myself, but if somebody wants to beat me to it the files are at.. http://macgeek.dyndns.org/empeg/ext3/ext3-0.0.3b.patch.gz http://macgeek.dyndns.org/empeg/ext3/empeg2b13-2.2.17.diff.gz http://macgeek.dyndns.org/empeg/ext3/mountHack.patch.gz http://macgeek.dyndns.org/empeg/ext3/v283.hijack.v200b13-2.2.17+ext3.patch.gz http://macgeek.dyndns.org/empeg/ext3/zImage-mk2-linux2.2.17-hj283-ext3-mh.gz http://macgeek.dyndns.org/empeg/ext3/e2fsck.gz It is a lot of work to maintain that many different patches. If you aren't compiling stuff yourself all you need is the last two and the instructions earlier in this thread. -Mike

All I have to say is WOAH NELLY! There's got to be an easier way for you to have all those archives ... Between you and Mr. Hijack, I bet you guys spend half your time gzipping and tarring files.

I appreciate you making it so easy for us to update this stuff. You *could* have just given us a diff and told us to compile our own stuff. Thanks a bunch!

Greg

Hmm, for some reason those didn't turn into links. I miss regular HTML. Weird, here they are again with explicit tags...

ext3 Patch
kernel patch to 2.2.17
Mount Hack
Updated Hijack Patch
Compiled Kernel
e2fsck

-Mike

All I have to say is WOAH NELLY! There's got to be an easier way for you to have all those archives ... Between you and Mr. Hijack, I bet you guys spend half your time gzipping and tarring files.

Nah, I spend half my time cross-compiling on a 400Mhz celeron. Now that is painful. Actually I have an ulterior motive in that I hope Mark will start including the 2.2.17 patch and the hijack changes as part of hijack so I don't have to maintain as much stuff. Then I would just be maintaining ext3 and the mount changes which would be a lot simpler.

I appreciate you making it so easy for us to update this stuff.

No problem, I had considered making an installer script for all of this, but I actually wanted to make it a little bit harder to keep it to just people who knew what they are doing until it is a little more proven.

-Mike

OK, I am successfully running b13 with the patched kernel linked above and ext3'd drives. There was only one minor glitch in the upgrade. I assumed that the stock empeg kernel would simply mount the drives as ext2. Instead it complained about "couldn't mount because of unsupported optional features" and did not mount the ext3 drives at all. After upgrading the kernel everything works fine.

So if anybody does change to ext3 and then wants to go back to ext2 you will need to delete the journal file and do an fsck to get the bits cleared so that the stock empeg kernel will mount your drives again.

-Mike

You should be able to mount it as ext2 though. So long as there is nothing that needs to be replayed from the journal. Very odd

- Trevor

Wow this sounds like a pretty easy upgrade... Are there any potential problems with using ext3? Maybe speed wise or something? It sounds like something that everyone should do, and maybe even something that should be considered for future inclusion in the Empeg standard distribution, no?

In reply to:

---

Are there any potential problems with using ext3?

---

I haven't hit any so far. It doesn't seem noticably slower to me. The biggest potential problem that I see is that this is a pretty out of date version of ext3 and I don't really know what kind of bugs may be lurking. I've got one bug that I just discovered. This does not work quite right on a dual drive empeg. When the player remounts the drives after a sync it throws a stack dump and exits. It only seems to happen on my dual drive player and I don't think it happens every time. I may just need a larger journal file or something since the second drive in my dual drive player is quite a bit larger (30 Gig vs 10) than the other drives I am working with. I am not up for fighting with it tonight though. -Mike

Run into a snag Mike.

I updated the empeg with the factory b13 update.

I updated the empeg with your precompiled zImage with hijack and whotnot.

I put e2fsck in the bin, did the moves of the old fsck and linked the names to the new e2fsck.

Now, the next part requires that I have mounted /drive0 and /drive1 ... which on any other occasion this would be true. But evidently upon boot up of the empeg, these did not mount. When I try to mount them with the -t ext2 ... the mount doesn't know what to do and tells me how I should be using the command line (usage) ... when I omit the -t ext2, it simply says there is no journal found on the drive.

What now boss?

Greg

My guess is that it got mounted as ext2 at some point during your upgrade and the info about the journal file got excised. Try remounting it with that ``journal='' syntax from above.

But I haven't even gotten to the point where I can build a journal file yet. In order to build it (at least from what I understand) I have to have it mounted first. And in order to mount it, it wants it to already have a journal file. Like I'm in some infinite loop of my own ignorance here. It's almost like dealing with the driver's license people. They want to see your ID before they'll allow you to get a replacement ID for the one you've lost, but you lost it, so you can't show it, so they won't give you a replacement. (Not quite, but the sentiment is there).

It seems like either I'm missing a step, or as soon as I update the kernel (with the required automatic restart), it thinks I should already have journal files. Is there a way to update the kernel by hand, without rebooting, so that I may build the journal files at that point before it unmounts them for good?

Greg

In reply to:

---

Run into a snag Mike.

---

You are a little out of order. You should create the journal files before updating the kernel. The kernel I provided will always change '-t ext2' into '-t ext3' so you wouldn't be able to mount as ext2 with that kernel. Grab the stock kernel off mlord's site so you can get the drives mounted as ext2 and then do the journal step before going back to my kernel. Once you have the journal files created figure out what inode they are at with 'ls -i' and then go back to my kernel and mount the drives with the 'journal=<inode>' option.

-Mike

In reply to:

---

It seems like either I'm missing a step

---

Actually I just reread my original post and I missed that step. I didn't specify when you should flash the kernel. Like I said in that other post you need to flash it after creating the journal files but before mounting as ext3. Sorry for the confusion.

The next time I release patches I will modify the mountHack patch so that you can mount the drives as ext2 on an alternate mount point with my kernel if necessary (I'll check the mount point instead of the dev name to determine if the type needs to be changed to ext3).

-Mike

Sorry. I assumed that you'd already done it once and were having trouble getting it back after the upgrade to b13

In case it isn't clear steps 3,4, and 5 from the original post are not necessary as the MountHack deals with all that. I'll throw together a web page with revised instructions when I have time.

-Mike

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails. This would allow ext3 to be part of hijack

Do we still need to do this?

5. You may need to replace the empegs init with a script that mounts the drives as ext3 instead of ext2. I already was using a custom script so I didn't worry about this.

I don't have a custom init script, if this isn't needed anymore with the new mount hacks in the latest kernel archive, then I think I'm up and running.

g

One last question ... should I add a journal file to the root file system? Currently I've got it on drive0 and drive1. The box complains about not having a journal file when mounting root, but remembers to mount it as ext2 instead.

Also does this new kernel almost support reiserfs? This was in the log:

Tried to mount /dev/hda4 as reiserfs but got error 19

Another note, while poking around ... /etc/mtab shows:
/dev/hda4 /drive0 ext3 rw,journal=12 0 0
/dev/hdc4 /drive1 ext3 rw,journal=12 0 0

Should these really be mounted rw? Is this so it can update the journal file while running?

Mike this is really exciting. I've been waiting for this for a long time, thanks a bunch for your efforts!

Greg

In reply to:

---

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails.

---

Yep, I agree. Feel free to submit a patch ;-)

In reply to:

---

Do we still need to do this?

5. You may need to replace the empegs init with a script that mounts the drives as ext3 instead of ext2. I already was using a custom script so I didn't worry about this.

---

Nope, you can skip steps 3,4, and 5. The kernel takes care of all that now. It sounds like you are all set. So you have a dual drive player right? Do you get a stack trace when the player remounts the drives after a sync? I still haven't had time to try and fix that one.

-Mike

In reply to:

---

One last question ... should I add a journal file to the root file system? Currently I've got it on drive0 and drive1. The box complains about not having a journal file when mounting root, but remembers to mount it as ext2 instead.

Also does this new kernel almost support reiserfs? This was in the log:

Tried to mount /dev/hda4 as reiserfs but got error 19

---

I haven't tried creating a journal file on any of the other partitions since they get overwritten with every upgrade and they aren't large enough for the fsck times to be an issue. You can safely ignore that message about not finding a journal.

The new kernel does not support reiserfs. I actually ripped out all the reiser code that the empeg guys had added in since some of it conflicts with ext3. The reason you get that error 19 message appears to be because the player or stock init attempts to mount as reiser first and then falls back to ext2. Should be safe to ignore it.

-Mike

In reply to:

---

Another note, while poking around ... /etc/mtab shows:
/dev/hda4 /drive0 ext3 rw,journal=12 0 0
/dev/hdc4 /drive1 ext3 rw,journal=12 0 0

Should these really be mounted rw? Is this so it can update the journal file while running?

---

AFAIK /etc/mtab is not used. The player explicitly mounts the drives as readonly. If you cat /proc/mounts you can see the current state of the drives which will be ro unless you have remounted them read-write.

-Mike

I didn't see a stack trace, but that doesn't mean there wasn't one. I'll say it wasn't obvious if there was. It looks good from here, and is working great.

Greg

In reply to:

---

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails. This would allow ext3 to be part of hijack

---

OK, ask and ye shall receive. The kernel here has been updated. It will now try to mount all ext2 partitions as ext3 and if that fails then fallback to ext2. No patch available yet (I need to get back to work).

With this kernel you no longer have to worry about the order you do the above steps in or what drives are what format. It should just do the write thing for you.

-Mike

OK, I have created a simple html page with a little better instructions and links to all the latest files here. I will continue to update that page with further changes as necessary. It also contains the latest mountHack patch that falls back to ext2 if necessary and an updated kernel binary/patches for hijack-288.

-Mike

Got it up and running just fine with the above kernel, thanks for the work you put into this.

Now that a fallback exists in the kernel, can we get this officially into Hijack? (Of course a patch will need to be sent to Mark with all this, but that shouldn't be too hard.)

To reverse this, all I need to do is revert to a normal kernel, and zap the .journal file right? The next boot in the car came up to a no drives found error, and I have a feeling it's drive0 dieing. I want to reverse all the custom changes though just to make sure. (It came back to life on it's own, I heard a drive noise in the car while I was stopped, pulled the unit and reinseted it to be welcomed to my music again).

In reply to:

---

To reverse this, all I need to do is revert to a normal kernel, and zap the .journal file right?

---

After you remove the journal file (or you can do a 'tunefs -O ^NO_JOURNAL /dev/hdxx') you will want to run a fsck to clear the journal flags it sets. Then you are back to stock form.

-Mike

In reply to:

---

Now that a fallback exists in the kernel, can we get this officially into Hijack? (Of course a patch will need to be sent to Mark with all this, but that shouldn't be too hard.)

---

All the patches needed to integrate into hijack should be linked on the site I posted above. It would be nice to get a few more people using it, including a person or two that is NOT using ext3 to make sure all the bugs are worked out before we subject the masses to it. Then again the masses have had a pretty cushy life lately, maybe they deserve a few bugs :-)

-Mike

I cannot see putting this into Hijack, sorry.

It takes up more RAM (no good for Mk1/Mk2 players, okay maybe for mk2a),

It renders the player incompatible with the standard software .upgrade tools for future betas/releases.

It would generate so much email to me ("It's broken..") that I just plain refuse to have anything to do with it.

But I like it.

Cheers

You may want to include this on your web site. I've been running it successfully with no hickups, since last we talked (yesterday or day before?)

Greg

Hmm, I've uninstalled ext3 and seen no problems running my empeg this morning. Last night I saw ide errors on the console indicating drive timeouts and read failures, with the last line indicating a reset of ide0 was done.

EXT3-fs error (device ide0(3,4)): ext3_readdir: bad entry in directory #2049: directory entry across blocks - offset=768, inode=536872065, rec_len=8208, name_len=5

The above is one of the other errors I saw, I did get this quite a bit (similar errors, but unfortunatly I didn't have them logged).

I'll continue to let it run today with no ext3, and try putting it on again tomorrow.

In reply to:

---

I cannot see putting this into Hijack, sorry.

It takes up more RAM (no good for Mk1/Mk2 players, okay maybe for mk2a),

It renders the player incompatible with the standard software .upgrade tools for future betas/releases.

It would generate so much email to me ("It's broken..") that I just plain refuse to have anything to do with it.

---

Sorry, but I have to argue with some of your points.

This doesn't take that much more memory, possibly because all the reiserfs code is removed. The kernel is 48k larger than a plain hijack kernel. Not bad for 3 kernel versions newer and support for an additional fs.

This does absolutely nothing unless the user chooses to use ext3. If they make that choice they have to know how to convert their drives and if they are smart enough to figure that out they should be smart enough to undo it if necessary. Unless they are using ext3 it is completely compatible with empeg .upgrade files.

The sad fact is that if this does not become a part of hijack I probably won't do many more updates. It is a lot of work to keep this up to date with your super-fast hijack release schedule and unless someone else wants to volunteer for that I can only see three outcomes. 1. it becomes part of hijack. 2. I stop tweaking hijack to support this so you get ext3 or hijack but not both. 3. I drop it all together :-(

-Mike

Reiserfs is already removed in standard Hijack kernels, so no savings there.

But you're right, we really don't want to toss this away. As I said earlier, I am most interested in the 2.2.17 kernel upgrade.

Keep at it for a few more weeks, and get some others to beta test for you, with or without ext3 enabled.

I'm off to the Sierra Nevada for some serious rockin' over the next couple of weeks, and when I get back we should merge this into the Hijack codebase. I expect I'll continue building Hijack without ext3 included, but so long as it's in the codebase maintenance becomes a non-issue. And anyone can just rebuild a reconfigured kernel to get the ext3 support.

You'll probably have to wake me up again after I return in mid-August.

Cheers

Thats great Mark. I certainly don't mind continuing to host a kernel build for ext3, it is the constant integration that would have become a nightmare for me. Have a good time climbing and I'll harass you about it again when you turn back up.

-Mike

In reply to:

---

You may want to include this on your web site. I've been running it successfully with no hickups, since last we talked (yesterday or day before?)

---

Cool. It is actually up there already. So we have one good experience and one not so good experience (Drakino). We need more testers...

-Mike

In reply to:

---

Last night I saw ide errors on the console indicating drive timeouts and read failures, with the last line indicating a reset of ide0 was done.

---

What are the specs on your empeg (drive size, model, number of drives) and are the drives mounted RO or RW when you see the errors? As I mentioned in a previous post I am still having one problem with this when the player remounts the drives after a sync, but it only is happening on my dual drive player and I haven't gotten to spend much time on it yet. Also, one of the drives in that player has always seemed kind of flaky.

Anyway, make sure you do an fsck or two as well since that error sounds kind of ominous.

-Mike

Perfect, thanks.

I went in and removed the journal and ran fsck. Because of the journal change, it did a full check on both drives, but passed with no problem. If I add the ext3 kernel back on, even without having ext3 partitions, this is what I see:

empeg-car bootstrap v1.02 20001106 (hugo@empeg.com)
If there is anyone present who wants to upgrade the flash, let them speak now,
or forever hold their peace...it seems not. Let fly the Penguins of Linux!

e000 v1.04
Copying kernel...
Calling linux kernel...
Uncompressing Linux...................................... done, booting the kernel.
Linux version 2.2.17-rmk5-np17-empeg51-kernel2.2.17-ext3-hijack-v288 (mcomb@grovepc) (gcc version 2.95.2 19991024 (release)) #1 Mon Jul 29 21:12:59 PDT 2002
Processor: Intel StrongARM-1100 revision 11
NetWinder Floating Point Emulator V0.94.1 (c) 1998 Corel Computer Corp.
empeg-car player (hardware revision 9, serial number 40103965)
Command line: mem=16m
Calibrating delay loop... 207.67 BogoMIPS
Memory: 14960k/16M available (1032k code, 20k reserved, 368k data, 4k init)
Dentry hash table entries: 2048 (order 2, 16k)
Buffer cache hash table entries: 16384 (order 4, 64k)
Page cache hash table entries: 4096 (order 2, 16k)
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
TCP: Hash tables configured (ehash 16384 bhash 16384)
IrDA (tm) Protocols for Linux-2.2 (Dag Brattli)
IrCOMM protocol (Dag Brattli)
Starting kswapd v 1.5 
SA1100 serial driver version 4.27 with no serial options enabled
ttyS00 at 0xf8010000 (irq = 15) is a SA1100 UART
ttyS01 at 0xf8050000 (irq = 17) is a SA1100 UART
ttyS02 at 0xf8030000 (irq = 16) is a SA1100 UART
Signature is 20706d65 'emp '
Found custom animation at offset 0x9a38c
empeg display initialised.
empeg dsp audio initialised
empeg dsp mixer initialised
empeg dsp initialised
empeg audio-in initialised, CS4231A revision a0
empeg remote control/panel button initialised.
empeg usb initialised, PDIUSBD12 id 1012
empeg state support initialised 0089/88c1 (save to d0005500).
empeg RDS driverinitialised
empeg power-pic driver initialised (first boot)
RAM disk driver initialized:  16 RAM disks of 4096K size
empegsinglechannel IDE
Probing primary interface...
hdb: FUJITSU MHL2300AT, ATA DISK drive
hda: FUJITSU MHL2300AT, ATA DISKdrive
hdb: FUJITSU MHL2300AT, ATA DISK drive
ide0 at 0x000-0x007,0x038 on irq 6
hda: FUJITSU MHL2300AT, 28615MB w/2048kB Cache,CHS=58140/16/63
hdb: FUJITSU MHL2300AT, 28615MB w/2048kB Cache, CHS=58140/16/63
empeg-flash driver initialized
smc chipid/revision 0x3349
smc9194.c:v0.12 03/06/96 by Erik Stahlman (erik@vt.edu)

SMC9194: SMC91C94(r:9) at 0x4008000 IRQ:7 INTF:TPMEM:6144b MAC 00:02:d7:28:0f:7d
Partition check:
 hda: hda1 < hda5 hda6 > hda2 hda3 hda4
 hdb: hdb1 < hdb5 hdb6 > hdb2 hdb3hdb4
RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 320 blocks [1 disk] into ram disk...|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.done.
ext3: No journal on filesystem on 01:00
EXT3-fs: get root inode failed
VFS:Mounted root (ext2 filesystem).
empeg-pump v0.03 (19980601)
Press Ctrl-A to enter pump...ext3: No journal on filesystem on03:05
EXT3-fs: get root inode failed
VFS: Mounted root (ext2 filesystem) readonly.
change_root: old root has d_count=1
Trying tounmount old root ... okay
Freeing unused kernel memory: 4k initempeg init 0.8
I see this is a developer image!
Mounting proc
Mounting first music partition
Tried to mount /dev/hda4 as reiserfs but got error 19changing mount type for:/dev/hda4

ext3:Nojournal on filesystem on 03:04
EXT3-fs: get root inode failed
ext3 mount of /dev/hda4 failed falling back toext2.
Mounting second music parchanging mount type for:/dev/hdc4
tition
ext3: No journal on filesystem on 03:44
EXT3-fs: get rootinode failed
ext3 mount of /dev/hdc4 failed falling back to ext2.
Remounting first music partition read-only
Remounting secondmusic partition read-only
Press 'q' now to go into development mode. You Have Zero Seconds To Comply...
Starting player

hijack: removed menu entry: "Hard Disk Detection"
khttpd: listening on port 80
kftpd: listening on port 21
Usingnon-standard cache size 118 (adjustment 16)
  player.cpp        : 385:empeg-car 2.00-beta13 2002/07/24.
hda:lost interrupt
hda: status error: status=0x58
hda: drive not ready for command
hda: status error: status=0x58
hda:drive not ready for command
hda: status timeout: status=0xd0
hda: drive not ready for command
ide0: reset:success
Prolux 4 empeg car - 2.1434 Jul 24 2002
Vcb: 0x4076d000
  serial_notify_thread.cpp: 116:@@ N2100
 serial_notify_thread.cpp: 117:@@ F0xf080
  serial_notify_thread.cpp: 118:@@ TWaiting for Daddy - Sam ...
  serial_notify_thread.cpp:119:@@ AMichael Kamen
  serial_notify_thread.cpp: 120:@@ GSoundtrack
  serial_notify_thread.cpp: 116:@@ N2100
  serial_notify_thread.cpp: 117:@@ F0xf080
  serial_notify_thread.cpp: 118:@@ TWaiting for Daddy - Sam ...
 serial_notify_thread.cpp: 119:@@ AMichael Kamen
  serial_notify_thread.cpp: 120:@@ GSoundtrack
  serial_notify_thread.cpp: 180:@@ #f080 0:00:13
  serial_notify_thread.cpp: 170:@@ S1
! mp3_decoder.cpp   :1049:Failed to find valid sync after seeking to offset 311099,error=0xc0044000
  serial_notify_thread.cpp: 180:@@ #f080  0:00:14

When the IDE errors are popping up, the boot screen shows the loading of each player component. It typicially freezes there on Loading Visuals or the database until the ide reset appears above. Working around with the player ling enough, and similar errors appear. The player ran for 20 hours playing with the normal Beta 13 Hijack kernel. During that time, I also did some minor drive writes to both, and never saw a single error.

Do you get any errors when using a kernel without the ext3 changes? If not then it's most likely there is a bug in the early versions of the ext3 FS code.

- Trevor

As I indicated loosly in the end of my last message: The problem goes away when I flash a normal hijack kernel on, and reappars the first boot with the ext3 kernel. So I tend to agree, some sort of glitch in ext3 is managing to cause this.

The scarry part was when I took my player out to the car after going to ext3 on Tuesday. It booted to the dead penguin and the error of "No drives detected". Powercycling it right away didn't help, but 5 mins later it came back, but with the above noted delay in the boot.

The above problem made me really think it was the hard drive failing, but considering it goes away with the normal kernel, well, I'm a bit puzzled.

Very interesting. I still think it is very possible that you have a flaky drive or maybe a bad cable and the reason you are seeing this is not because of ext3, but because of the newer kernel being more sensitive to a hardware issue. But of course I could be completely wrong If you are up for a little testing try this kernel which is patched up to 2.2.17 but does not contain ext3 support or hijack. Might help us narrow down the problem a little at least.

-Mike

Tried the 2.2.17 kernel linked above, and it worked fine. Flashed the ext 3 kernel again, and the same boot log as above. Tried ti twice more of switching back and forth, the errors and pause in boot only occur when ext3 support is in the kernel. All the time never actually moving the empeg, so no issues where the cable was just providing a coincidental issue.

*boggle*

In reply to:

---

Tried the 2.2.17 kernel linked above, and it worked fine. Flashed the ext 3 kernel again, and the same boot log as above.

---

Huh. I don't know what to tell you. It would be nice if we had the linux IDE guru around to maybe help translate that. Mark, are you still here? It doesn't surprise me that there are bugs in that ext3 version, but that one shows itself in a weird way. I did a quick google search for some of that error and most of the results I found suggest dieing hardware.

I had planned to try to get ext3 a little more current when I have some time (maybe this weekend). Hopefully that will help.

-Mike

One possibility is a bad sector where the journal file is - ext2 wouldn't have a problem, because it wouldn't be touching that sector, but ext3 would.

In reply to:

---

One possibility is a bad sector where the journal file is - ext2 wouldn't have a problem, because it wouldn't be touching that sector, but ext3 would.

---

Sounds good in theory, but he doesn't even have a journal file set up at the moment if I understand correctly. I suppose it could still be looking for one when it tries to do the ext3 mount before falling back to ext2...

Well I've been up for a little over two weeks no with no issues. Today, however, while doing a moderately large sync emplode has decided to do a "Checking Media -- Checking disk integrity" ... for some reason this is taking a long time. It's a 10+30 gig configuration. I don't remember it ever taking more than 10 minutes to do this check previously. Currently I'm at the 15 minute mark. The weird thing is that the previous sync was normal, and I haven't crashed the player in a long time.

Do you think this has anything to do with ext3? And shouldn't it go super fast with the new ext3 savvy fsck?

Greg

In reply to:

---

Do you think this has anything to do with ext3? And shouldn't it go super fast with the new ext3 savvy fsck?

---

Actually an ext3 fsck won't be any faster. The thing is it should never have to do an fsck in the first place as it fixes any file system errors when the volume is first mounted. The only reason you are even getting an fsck is because the player software explicitly calls it. IIRC hijack has an option to disable the periodic fscks that you are seeing. I haven't actually tried it myself though.

FWIW, on my 40 Gig empeg fscks always seem to take at least a half hour. Did it eventually finish OK?

BTW, if anybody is wondering why I haven't touched this in a while it is mainly because I am in the middle of moving so haven't had much free time to work on it. I am still hoping to get it patched up a bit farther to (hopefully) fix Drakino's problem and the remount problem I have. I probably won't be able to get to it until next month though as I won't have internet access (sacrilegious!) for a bit after finishing the move into the new place.

-Mike

The reason for the sudden filesystem check is that you have not yet done this:


tune2fs -c0 -i0 /dev/hdXX (replace XX with appropriate device suffix for the filesystem in question, eg. /dev/hda5 ).

Ahh .. you're right, I haven't done that. I will do it now though, thanks!

Greg

Mike (Comb) is this already in your doc for setup?

tune2fs -c0 -i0 /dev/hdXX

Mike (Comb) is this already in your doc for setup?

tune2fs -c0 -i0 /dev/hdXX

It is mentioned briefly in the last paragraph of the website but I don't give details. I haven't checked if the version of tune2fs on the empeg is new enough to set that flag. Did it work OK for you? If so I will add it to the doc.

-Mike

The reason for the sudden filesystem check is that you have not yet done this:


tune2fs -c0 -i0 /dev/hdXX (replace XX with appropriate device suffix for the filesystem in question, eg. /dev/hda5 ).

It appears that there is an option in hijack to bypass that check as well, but I tried it and it didn't work for me (the drives are still checked after 20 or so mounts). Could it be broken or does it not like ext3?

Oh, and welcome back, how was the climbing?

-Mike

I unmounted the drives, ran that command and it said something about ext3 being updated, so I assume it worked like it should. At the very least it didn't die or complain.

greg

I unmounted the drives, ran that command and it said something about ext3 being updated, so I assume it worked like it should. At the very least it didn't die or complain.

Sounds good. I'll update my page and try it myself once I get my network put back together in my new apartment. Oh, by the way, that ext3 page will be down for at least a week starting sometime saturday afternoon while I wait to get internet access at the new place, so if anybody wants any files grab them now.

-Mike

I am still hoping to get it patched up a bit farther to (hopefully) fix Drakino's problem

I finally began seeing other signs of problems with my drive 1, so I have contacted support about it. It seems ext3 just had a way of seeing the future problem :-)

It's to the point now where I get garbled music playback at times from data being read off the drive wrong. I haven't run ext3 since I last posted. so thats not causing the problem as far as I can tell.