In case it isn't clear steps 3,4, and 5 from the original post are not necessary as the MountHack deals with all that. I'll throw together a web page with revised instructions when I have time.

-Mike

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails. This would allow ext3 to be part of hijack

Do we still need to do this?

5. You may need to replace the empegs init with a script that mounts the drives as ext3 instead of ext2. I already was using a custom script so I didn't worry about this.

I don't have a custom init script, if this isn't needed anymore with the new mount hacks in the latest kernel archive, then I think I'm up and running.

g

One last question ... should I add a journal file to the root file system? Currently I've got it on drive0 and drive1. The box complains about not having a journal file when mounting root, but remembers to mount it as ext2 instead.

Also does this new kernel almost support reiserfs? This was in the log:

Tried to mount /dev/hda4 as reiserfs but got error 19

Another note, while poking around ... /etc/mtab shows:
/dev/hda4 /drive0 ext3 rw,journal=12 0 0
/dev/hdc4 /drive1 ext3 rw,journal=12 0 0

Should these really be mounted rw? Is this so it can update the journal file while running?

Mike this is really exciting. I've been waiting for this for a long time, thanks a bunch for your efforts!

Greg

In reply to:

---

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails.

---

Yep, I agree. Feel free to submit a patch ;-)

In reply to:

---

Do we still need to do this?

5. You may need to replace the empegs init with a script that mounts the drives as ext3 instead of ext2. I already was using a custom script so I didn't worry about this.

---

Nope, you can skip steps 3,4, and 5. The kernel takes care of all that now. It sounds like you are all set. So you have a dual drive player right? Do you get a stack trace when the player remounts the drives after a sync? I still haven't had time to try and fix that one.

-Mike

In reply to:

---

One last question ... should I add a journal file to the root file system? Currently I've got it on drive0 and drive1. The box complains about not having a journal file when mounting root, but remembers to mount it as ext2 instead.

Also does this new kernel almost support reiserfs? This was in the log:

Tried to mount /dev/hda4 as reiserfs but got error 19

---

I haven't tried creating a journal file on any of the other partitions since they get overwritten with every upgrade and they aren't large enough for the fsck times to be an issue. You can safely ignore that message about not finding a journal.

The new kernel does not support reiserfs. I actually ripped out all the reiser code that the empeg guys had added in since some of it conflicts with ext3. The reason you get that error 19 message appears to be because the player or stock init attempts to mount as reiser first and then falls back to ext2. Should be safe to ignore it.

-Mike

In reply to:

---

Another note, while poking around ... /etc/mtab shows:
/dev/hda4 /drive0 ext3 rw,journal=12 0 0
/dev/hdc4 /drive1 ext3 rw,journal=12 0 0

Should these really be mounted rw? Is this so it can update the journal file while running?

---

AFAIK /etc/mtab is not used. The player explicitly mounts the drives as readonly. If you cat /proc/mounts you can see the current state of the drives which will be ro unless you have remounted them read-write.

-Mike

I didn't see a stack trace, but that doesn't mean there wasn't one. I'll say it wasn't obvious if there was. It looks good from here, and is working great.

Greg

In reply to:

---

For this to be a painless upgrade, the kernel really needs to try mounting as ext2 if the ext3 mount fails. This would allow ext3 to be part of hijack

---

OK, ask and ye shall receive. The kernel here has been updated. It will now try to mount all ext2 partitions as ext3 and if that fails then fallback to ext2. No patch available yet (I need to get back to work).

With this kernel you no longer have to worry about the order you do the above steps in or what drives are what format. It should just do the write thing for you.

-Mike

OK, I have created a simple html page with a little better instructions and links to all the latest files here. I will continue to update that page with further changes as necessary. It also contains the latest mountHack patch that falls back to ext2 if necessary and an updated kernel binary/patches for hijack-288.

-Mike

Got it up and running just fine with the above kernel, thanks for the work you put into this.

Now that a fallback exists in the kernel, can we get this officially into Hijack? (Of course a patch will need to be sent to Mark with all this, but that shouldn't be too hard.)

To reverse this, all I need to do is revert to a normal kernel, and zap the .journal file right? The next boot in the car came up to a no drives found error, and I have a feeling it's drive0 dieing. I want to reverse all the custom changes though just to make sure. (It came back to life on it's own, I heard a drive noise in the car while I was stopped, pulled the unit and reinseted it to be welcomed to my music again).

In reply to:

---

To reverse this, all I need to do is revert to a normal kernel, and zap the .journal file right?

---

After you remove the journal file (or you can do a 'tunefs -O ^NO_JOURNAL /dev/hdxx') you will want to run a fsck to clear the journal flags it sets. Then you are back to stock form.

-Mike

In reply to:

---

Now that a fallback exists in the kernel, can we get this officially into Hijack? (Of course a patch will need to be sent to Mark with all this, but that shouldn't be too hard.)

---

All the patches needed to integrate into hijack should be linked on the site I posted above. It would be nice to get a few more people using it, including a person or two that is NOT using ext3 to make sure all the bugs are worked out before we subject the masses to it. Then again the masses have had a pretty cushy life lately, maybe they deserve a few bugs :-)

-Mike

I cannot see putting this into Hijack, sorry.

It takes up more RAM (no good for Mk1/Mk2 players, okay maybe for mk2a),

It renders the player incompatible with the standard software .upgrade tools for future betas/releases.

It would generate so much email to me ("It's broken..") that I just plain refuse to have anything to do with it.

But I like it.

Cheers

You may want to include this on your web site. I've been running it successfully with no hickups, since last we talked (yesterday or day before?)

Greg

Hmm, I've uninstalled ext3 and seen no problems running my empeg this morning. Last night I saw ide errors on the console indicating drive timeouts and read failures, with the last line indicating a reset of ide0 was done.

EXT3-fs error (device ide0(3,4)): ext3_readdir: bad entry in directory #2049: directory entry across blocks - offset=768, inode=536872065, rec_len=8208, name_len=5

The above is one of the other errors I saw, I did get this quite a bit (similar errors, but unfortunatly I didn't have them logged).

I'll continue to let it run today with no ext3, and try putting it on again tomorrow.

In reply to:

---

I cannot see putting this into Hijack, sorry.

It takes up more RAM (no good for Mk1/Mk2 players, okay maybe for mk2a),

It renders the player incompatible with the standard software .upgrade tools for future betas/releases.

It would generate so much email to me ("It's broken..") that I just plain refuse to have anything to do with it.

---

Sorry, but I have to argue with some of your points.

This doesn't take that much more memory, possibly because all the reiserfs code is removed. The kernel is 48k larger than a plain hijack kernel. Not bad for 3 kernel versions newer and support for an additional fs.

This does absolutely nothing unless the user chooses to use ext3. If they make that choice they have to know how to convert their drives and if they are smart enough to figure that out they should be smart enough to undo it if necessary. Unless they are using ext3 it is completely compatible with empeg .upgrade files.

The sad fact is that if this does not become a part of hijack I probably won't do many more updates. It is a lot of work to keep this up to date with your super-fast hijack release schedule and unless someone else wants to volunteer for that I can only see three outcomes. 1. it becomes part of hijack. 2. I stop tweaking hijack to support this so you get ext3 or hijack but not both. 3. I drop it all together :-(

-Mike

Reiserfs is already removed in standard Hijack kernels, so no savings there.

But you're right, we really don't want to toss this away. As I said earlier, I am most interested in the 2.2.17 kernel upgrade.

Keep at it for a few more weeks, and get some others to beta test for you, with or without ext3 enabled.

I'm off to the Sierra Nevada for some serious rockin' over the next couple of weeks, and when I get back we should merge this into the Hijack codebase. I expect I'll continue building Hijack without ext3 included, but so long as it's in the codebase maintenance becomes a non-issue. And anyone can just rebuild a reconfigured kernel to get the ext3 support.

You'll probably have to wake me up again after I return in mid-August.

Cheers

Thats great Mark. I certainly don't mind continuing to host a kernel build for ext3, it is the constant integration that would have become a nightmare for me. Have a good time climbing and I'll harass you about it again when you turn back up.

-Mike

In reply to:

---

You may want to include this on your web site. I've been running it successfully with no hickups, since last we talked (yesterday or day before?)

---

Cool. It is actually up there already. So we have one good experience and one not so good experience (Drakino). We need more testers...

-Mike

In reply to:

---

Last night I saw ide errors on the console indicating drive timeouts and read failures, with the last line indicating a reset of ide0 was done.

---

What are the specs on your empeg (drive size, model, number of drives) and are the drives mounted RO or RW when you see the errors? As I mentioned in a previous post I am still having one problem with this when the player remounts the drives after a sync, but it only is happening on my dual drive player and I haven't gotten to spend much time on it yet. Also, one of the drives in that player has always seemed kind of flaky.

Anyway, make sure you do an fsck or two as well since that error sounds kind of ominous.

-Mike

Perfect, thanks.

I went in and removed the journal and ran fsck. Because of the journal change, it did a full check on both drives, but passed with no problem. If I add the ext3 kernel back on, even without having ext3 partitions, this is what I see:

empeg-car bootstrap v1.02 20001106 (hugo@empeg.com)
If there is anyone present who wants to upgrade the flash, let them speak now,
or forever hold their peace...it seems not. Let fly the Penguins of Linux!

e000 v1.04
Copying kernel...
Calling linux kernel...
Uncompressing Linux...................................... done, booting the kernel.
Linux version 2.2.17-rmk5-np17-empeg51-kernel2.2.17-ext3-hijack-v288 (mcomb@grovepc) (gcc version 2.95.2 19991024 (release)) #1 Mon Jul 29 21:12:59 PDT 2002
Processor: Intel StrongARM-1100 revision 11
NetWinder Floating Point Emulator V0.94.1 (c) 1998 Corel Computer Corp.
empeg-car player (hardware revision 9, serial number 40103965)
Command line: mem=16m
Calibrating delay loop... 207.67 BogoMIPS
Memory: 14960k/16M available (1032k code, 20k reserved, 368k data, 4k init)
Dentry hash table entries: 2048 (order 2, 16k)
Buffer cache hash table entries: 16384 (order 4, 64k)
Page cache hash table entries: 4096 (order 2, 16k)
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
TCP: Hash tables configured (ehash 16384 bhash 16384)
IrDA (tm) Protocols for Linux-2.2 (Dag Brattli)
IrCOMM protocol (Dag Brattli)
Starting kswapd v 1.5 
SA1100 serial driver version 4.27 with no serial options enabled
ttyS00 at 0xf8010000 (irq = 15) is a SA1100 UART
ttyS01 at 0xf8050000 (irq = 17) is a SA1100 UART
ttyS02 at 0xf8030000 (irq = 16) is a SA1100 UART
Signature is 20706d65 'emp '
Found custom animation at offset 0x9a38c
empeg display initialised.
empeg dsp audio initialised
empeg dsp mixer initialised
empeg dsp initialised
empeg audio-in initialised, CS4231A revision a0
empeg remote control/panel button initialised.
empeg usb initialised, PDIUSBD12 id 1012
empeg state support initialised 0089/88c1 (save to d0005500).
empeg RDS driverinitialised
empeg power-pic driver initialised (first boot)
RAM disk driver initialized:  16 RAM disks of 4096K size
empegsinglechannel IDE
Probing primary interface...
hdb: FUJITSU MHL2300AT, ATA DISK drive
hda: FUJITSU MHL2300AT, ATA DISKdrive
hdb: FUJITSU MHL2300AT, ATA DISK drive
ide0 at 0x000-0x007,0x038 on irq 6
hda: FUJITSU MHL2300AT, 28615MB w/2048kB Cache,CHS=58140/16/63
hdb: FUJITSU MHL2300AT, 28615MB w/2048kB Cache, CHS=58140/16/63
empeg-flash driver initialized
smc chipid/revision 0x3349
smc9194.c:v0.12 03/06/96 by Erik Stahlman (erik@vt.edu)

SMC9194: SMC91C94(r:9) at 0x4008000 IRQ:7 INTF:TPMEM:6144b MAC 00:02:d7:28:0f:7d
Partition check:
 hda: hda1 < hda5 hda6 > hda2 hda3 hda4
 hdb: hdb1 < hdb5 hdb6 > hdb2 hdb3hdb4
RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 320 blocks [1 disk] into ram disk...|./.-.\.|./.-.\.|./.-.\.|./.-.\.|./.-.\.done.
ext3: No journal on filesystem on 01:00
EXT3-fs: get root inode failed
VFS:Mounted root (ext2 filesystem).
empeg-pump v0.03 (19980601)
Press Ctrl-A to enter pump...ext3: No journal on filesystem on03:05
EXT3-fs: get root inode failed
VFS: Mounted root (ext2 filesystem) readonly.
change_root: old root has d_count=1
Trying tounmount old root ... okay
Freeing unused kernel memory: 4k initempeg init 0.8
I see this is a developer image!
Mounting proc
Mounting first music partition
Tried to mount /dev/hda4 as reiserfs but got error 19changing mount type for:/dev/hda4

ext3:Nojournal on filesystem on 03:04
EXT3-fs: get root inode failed
ext3 mount of /dev/hda4 failed falling back toext2.
Mounting second music parchanging mount type for:/dev/hdc4
tition
ext3: No journal on filesystem on 03:44
EXT3-fs: get rootinode failed
ext3 mount of /dev/hdc4 failed falling back to ext2.
Remounting first music partition read-only
Remounting secondmusic partition read-only
Press 'q' now to go into development mode. You Have Zero Seconds To Comply...
Starting player

hijack: removed menu entry: "Hard Disk Detection"
khttpd: listening on port 80
kftpd: listening on port 21
Usingnon-standard cache size 118 (adjustment 16)
  player.cpp        : 385:empeg-car 2.00-beta13 2002/07/24.
hda:lost interrupt
hda: status error: status=0x58
hda: drive not ready for command
hda: status error: status=0x58
hda:drive not ready for command
hda: status timeout: status=0xd0
hda: drive not ready for command
ide0: reset:success
Prolux 4 empeg car - 2.1434 Jul 24 2002
Vcb: 0x4076d000
  serial_notify_thread.cpp: 116:@@ N2100
 serial_notify_thread.cpp: 117:@@ F0xf080
  serial_notify_thread.cpp: 118:@@ TWaiting for Daddy - Sam ...
  serial_notify_thread.cpp:119:@@ AMichael Kamen
  serial_notify_thread.cpp: 120:@@ GSoundtrack
  serial_notify_thread.cpp: 116:@@ N2100
  serial_notify_thread.cpp: 117:@@ F0xf080
  serial_notify_thread.cpp: 118:@@ TWaiting for Daddy - Sam ...
 serial_notify_thread.cpp: 119:@@ AMichael Kamen
  serial_notify_thread.cpp: 120:@@ GSoundtrack
  serial_notify_thread.cpp: 180:@@ #f080 0:00:13
  serial_notify_thread.cpp: 170:@@ S1
! mp3_decoder.cpp   :1049:Failed to find valid sync after seeking to offset 311099,error=0xc0044000
  serial_notify_thread.cpp: 180:@@ #f080  0:00:14

When the IDE errors are popping up, the boot screen shows the loading of each player component. It typicially freezes there on Loading Visuals or the database until the ide reset appears above. Working around with the player ling enough, and similar errors appear. The player ran for 20 hours playing with the normal Beta 13 Hijack kernel. During that time, I also did some minor drive writes to both, and never saw a single error.

Do you get any errors when using a kernel without the ext3 changes? If not then it's most likely there is a bug in the early versions of the ext3 FS code.

- Trevor

As I indicated loosly in the end of my last message: The problem goes away when I flash a normal hijack kernel on, and reappars the first boot with the ext3 kernel. So I tend to agree, some sort of glitch in ext3 is managing to cause this.

The scarry part was when I took my player out to the car after going to ext3 on Tuesday. It booted to the dead penguin and the error of "No drives detected". Powercycling it right away didn't help, but 5 mins later it came back, but with the above noted delay in the boot.

The above problem made me really think it was the hard drive failing, but considering it goes away with the normal kernel, well, I'm a bit puzzled.

Very interesting. I still think it is very possible that you have a flaky drive or maybe a bad cable and the reason you are seeing this is not because of ext3, but because of the newer kernel being more sensitive to a hardware issue. But of course I could be completely wrong If you are up for a little testing try this kernel which is patched up to 2.2.17 but does not contain ext3 support or hijack. Might help us narrow down the problem a little at least.

-Mike

Tried the 2.2.17 kernel linked above, and it worked fine. Flashed the ext 3 kernel again, and the same boot log as above. Tried ti twice more of switching back and forth, the errors and pause in boot only occur when ext3 support is in the kernel. All the time never actually moving the empeg, so no issues where the cable was just providing a coincidental issue.

*boggle*