Okay, a number of people, including His Esteemed Faqness, have reported temporary IR lockups (lasting 10-20 minutes) on their players in recent betas.

Unfortunately, it never seems to happen for me, and I'm one of the few around here who could diagnose one if it did happen. Darn.

But.. based on Tony's lucky coicidence of having such a lockup with a Hijack PopUp message at the same time, I can only conclude that the cause, as highly improbable as it seems (totally friggin impossible), is that the O/S "jiffies" counter must be stopping for whatever reason.

So.. As of Hijack v221 (now available), there is code within Hijack to detect such an occurance, and to display a pair of suitable messages on the screen.

Please, everyone, install this update, and report to me (both on the BBS and directly if you ever see one/both of these messages on the display of your RioCar/Empeg:

    ERROR: stuck jiffies

or

    unstuck at 500.34 secs   (any numeric value is fine)

Also don't forget to say which version of player software was installed, eg. v2.00-beta11.

Thanks

thank you mark.. your a pal.. i just wanna note... that it only seems to happen when its in "ac/home mode" never when its in car.. if that helps your any or confuses you more.

not true. Locks up in the car all the time for me and others. =]

Mark, i upgraded the kernel and i've got it playing and will all day until it locks up. Hasn't happened yet though.

I have seen it happen in both home and car modes, so that's not a data point.

And Mark, there's no need to refer to me in such a formal tone. "Sir FAQ Master" is plenty.

lol.. well i guess thats just my situation.. i always foced dc mode to have it not lock up on me.. and the good news is ive had the player running for the past 30min and no lock up... yippie.. so ill see how it goes

Sir FAQ Master, and other esteemed viewers, I may have found a cause of the lockups. I was tracking down the "jiffies" issue, and noticed a Hugo-ism in the kernel timer code, where he has some code which he says must run uninterruptable to reset the jiffie timer.. but the code fails to mask FIRQ's.

It's a single-character fix, and I'll put it out in v222 later today.

Of course, this is all just theory, so.. from v222 onwards, if you ever experience a lockup with Hijack, let me know.

In the meanwhile, does anyone recall exactly what FIRQs are used for on the RioCar (Mk2a)?

-ml

*crosses fingers*

ok mark player locked up.. and no message or hijack saving.. anything you want me to do or information to gain while its in this locked state.. as i said before i can still access the hijack menu.. but the player itself it locked up

Mmm.. that's different from Faq Master's lockup -- your jiffies are still working.

Point your browser at http://your.empeg/proc/interrupts
and refresh a few times: see if you can identify whether
or not "timer" is incrementing (yes), and SMC9194, and rdsirq
and any others. List them for me (or just save two successive
snapshots of the page).

Also look at empeg_ir and verify that it updates when you press on the remote.

-ml

Also, a few successive samples from /proc/stat might be useful.

-ml

ir codes work.. but the ftp/http dose not

ok heres what i got i kept hitting refresh till it froze again..

If your IR codes are working, then that's different lockup that the one we're looking for i believe.

Two of those displays, one right after the other, would be useful.

One by itself is not really.

Cheers

alrighty.. ill try and repeat it again

one erroe i notice when booting the player is this

Tried to mount /dev/hda4 as reiserfs but got error 19 coudl that have anything to do with it?

No, that message is just "normal drivel".

I assume the bug y'all are talking about is where all input is locked up? Lately I've been having this problem: When I hold a button on the remote -such as search, volume, equalizer, info, etc.- the player does nothing. But when I let go of the button it acts if I had just pressed it once instead of holding it down. It doesn't do it everytime. Perhaps my remote battery is dying? I hope it's not something wrong with the player.

ok mark i have a series of 4 messages.. in order
------Part1------
0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 34020 empeg_rdsirq
4: 199 empeg_input
6: 5499 ide0
7: 97 SMC9194
17: 147 serial
20: 1102 audio-empeg
26: 2923 timer
FIQ: cs4231dma

------Part2------
0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 49230 empeg_rdsirq
4: 199 empeg_input
6: 5499 ide0
7: 142 SMC9194
17: 147 serial
20: 1592 audio-empeg
26: 4202 timer
FIQ: cs4231dma

------Part3------
0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 67029 empeg_rdsirq
4: 199 empeg_input
6: 5499 ide0
7: 301 SMC9194
17: 147 serial
20: 2165 audio-empeg
26: 5699 timer
FIQ: cs4231dma

------Part4------
0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 94465 empeg_rdsirq
4: 253 empeg_input
6: 5500 ide0
7: 504 SMC9194
17: 147 serial
20: 3048 audio-empeg
26: 8006 timer
FIQ: cs4231dma

and the player crashed in the matter of seconds after the last one.

Nothing I can do for that one. Different issue from the "button lockup issue".

But if I worked at SonicBlue, and could reproduce the problem, I'd use Hijack khttpd to surf to /proc/nnn/ and examine what each of the player threads was doing, to see which one was stuck in a loop.

But that would require player source & map files.

Cheers

so tis kidna hopeless.. its pretty hard to reproduce the problem.. it jsut kinda dose it from tiem to time.. but like i said its only done it in "home/ac power" so my solutions is jsut to run it in car mode.. and everything is fine.. the car is the mode im more concerned about anyways.. i only bring it home for hacking and laoding music.. not for really using it as an alternative player..

Ok, my player has a unstuck at 0.0 Secs displayed on it. Im running 2.0 beta 11.

The clock and counters appear to be incrementing ok right now. Also, I noticed my clock has fallen 18 minutes behind since i set it 2 or 3 days ago.

Sean

http://empeg/proc/interrupts

0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 16809834 empeg_rdsirq
4: 3244 empeg_input
6: 42016 ide0
7: 1773 SMC9194
17: 146 serial
20: 540707 audio-empeg
26: 1296006 timer
FIQ: cs4231dma

0: 0 empeg_state
1: 0 empeg_usbirq
2: 0 empeg_cs4231
3: 16828466 empeg_rdsirq
4: 3244 empeg_input
6: 42016 ide0
7: 1794 SMC9194
17: 146 serial
20: 541307 audio-empeg
26: 1297572 timer
FIQ: cs4231dma






http://empeg/proc/stat

cpu 381874 0 39154 879257
disk 2112 1456 0 0
disk_rio 2089 1456 0 0
disk_wio 23 0 0 0
disk_rblk 4178 2912 0 0
disk_wblk 46 0 0 0
page 131097 384
swap 0 0
intr 18750577 0 0 0 16860729 3244 0 42016 1804 0 0 0 0 0 0 0 8 0 146 0 0 542345 0 0 0 0 0 1300285 0 0 0 0 0
ctxt 1508455
btime 1014914061
processes 159


cpu 382700 0 39265 879983
disk 2112 1456 0 0
disk_rio 2089 1456 0 0
disk_wio 23 0 0 0
disk_rblk 4178 2912 0 0
disk_wblk 46 0 0 0
page 131097 384
swap 0 0
intr 18772672 0 0 0 16880513 3244 0 42017 1814 0 0 0 0 0 0 0 8 0 146 0 0 542982 0 0 0 0 0 1301948 0 0 0 0 0
ctxt 1510454
btime 1014914061
processes 160

Okay, that's cool. Once the "unstuck" message appears, the problem has already gone away again.

And I guess it will always say "0.0 secs", so I'll change that message next time to show something more useful.

But the fact that you even saw it, indicates that jiffies took a pause somewhere -- kernel still running, but clock interrupts (for scheduler) not working, briefly.

That's the bug that might go away when I release my fix for Hugo's timer interrupt routine.

Cheers

Speaking of which, does the clock get corrected on reboot?

(while running, the Hijack clock display is from the kernel, not the clock hardware.. if the jiffies bug occurs, the kernel clock value will lag, but the clock hardware will continue to maintain reasonably accurate time..)

Very interesting! Yes it did correct the clock on reboot.

Mark, may I suggest you NOT fix Hugo's bug until you know for sure it's the cause of the problem? This problem seems to be intermittent, so it's a good idea to give people a chance to duplicate it. It will happen on my player at least once every two days, so I'll install a new Hijack tonight. Unfortunately it tens to happen when I'm in the car (I don't play tunes much at night unless I'm working on something player-related).

I'm sure you'll have a lot of feedback in this thread by tomorrow night.

Bruno

In reply to:

---

When I hold a button on the remote -such as search, volume, equalizer, info, etc.- the player does nothing. But when I let go of the button it acts if I had just pressed it once instead of holding it down. It doesn't do it everytime. Perhaps my remote battery is dying?

---

I sometimes get this on the front panel, normally only when the disk-access icon is up. Don't know if it's related, but it's an extra datum.

It's a single-character fix, and I'll put it out in v222 later today.

Got a diff for just this fix?

In the meanwhile, does anyone recall exactly what FIRQs are used for on the RioCar (Mk2a)?

Um, I thought ARMs have IRQs and FIQs, which makes it hard to know what the FIRQ you're talking about ...

Peter

Duh.. stuck midway between x86 and ARM these days.. Yeah, I meant FIQ. The fix is to just change the save_flags_cli() call into a save_flags_clif() call in the reset_timer() routine in the kernel. Hugo already knows about, too.

Cheers

>may I suggest you NOT fix Hugo's bug until you know for sure
>it's the cause of the problem?

Yeah, I'm thinking that too.

But there is no question that it's a real bug, and also that it causes at least some of the lockups. The buggy code executes 100 times/second, and only needs to be hit once with a simultaneous FIQ to be triggered.

Remember the lockups we had when I first added the temperature-reading code into Hijack? Those happened reasonably often, and were due to exactly the same coding problem. Except this new bug has a about 200 times the probability of occuring.

Most of the time, it will be barely noticeable, but some of the time it could lock up the player for as long as 19 minutes or so (32-bit timer at 3.6864mhz) -- which is pretty much what people are reporting here.

But a little more data would be nice to confirm all of this. So, I'll put out v222 this morning, WITHOUT the fix, but with more data collection for when it happens.

Cheers

Most of the time, it will be barely noticeable, but some of the time it could lock up the player for as long as 19 minutes or so (32-bit timer at 3.6864mhz) -- which is pretty much what people are reporting here.

Yeah, it is a bit spooky how everyone whose clock lurches, lurches by the same amount.

But a little more data would be nice to confirm all of this. So, I'll put out v222 this morning, WITHOUT the fix, but with more data collection for when it happens.

By the way, we ought to thank you really for all the work you're putting in to help us debug our own code and fix what's probably the worst outstanding bug in the 2.0 betas. So, thanks!

Peter

Here's the question.... Is the code you're fixing NEW in the 2.0x betas? Because there were no reported lockups back in the 1.03 days... So I would think that whatever the real bug is has to be in code that was introduced for 2.0... Right?

Dunno, but this particular bug has probably been there all along. I wonder if the frequency of FIQs has increased since the 1.03 stream?

Something like this gets influenced by all sorts of subtle behaviours in process priorities and whatnot, and it is entirely possible for the triggering interval to have been much lower in the past. Or not.

Also, does it just seem more common now, with about triple the number of player owners out there and posting to this BBS, happily coinciding with the v2 betas, or did it REALLY never occur before?

Remember, it probably happens a lot more than we think, cuz we've gotta be pushing buttons to notice it.

???

Actually, the FIQ's have a lot to do with the presence of a Tuner module.. I'm not that familiar with all of this, but my understanding is that without a Tuner they are less common (or not at all)? And the v2 betas began not long after Tuner modules began shipping, so there wasn't a lot of overlap time with v1.03 and Tuners (and FIQs?).

And without FIQs, the bug I'm looking at won't occur.

So.. what are FIQs used for, anyway?

Eh... I dunno, I agree that that the 2.0 betas coincided with a massive increase in the population, but we had a pretty large sample size to begin with during the 1.x series, and I don't think there were *ever* any reports of any sort of lockups of this nature. That was a long period of time, too... However, the other possibilities you mention (higher frequency of FIQs, changes in various timings/processes, etc) are all quite plausible. I'm 100% in favor of any effort to try to fix this problem, and this sounds promising. The lockups and the buggy info mode setting are the only two things that make running 2.0b11 painful for me, so solving this problem would be a huge win.

The price drop and 2.00beta3 both happened in December right? That is when the membership skyrocketed AND 2.00beta hit. I remember making a post along those lines back then.

Okay, Hijack v222 is out.

slightly improved "lockup messages". These now say stuck jiffies 5 or unstuck jiffies 8. The numbers will change, but they provide important information. Please report them here if you ever see either of these messages.

the Reboot Machine menu item now remounts all drives read-only before rebooting. This has been missing for a while, and I finally remembered to implement it. The remote "REBOOT" command (FTP,HTTP,/proc) has had this since day-1.

the bugfix for the jiffies reset is not included yet, but it will be in v223 later on. I suspect this will fix nearly all player lockups, but we'd like to collect a little more data first.

Upcoming features (sometime in March'2000):

ir_translate capability for Sony Stalk (this one's been simmering for a while, as I'm still not 100% sure how to present it, and have been distracted by the barbarians pounding on the gates).

The jiffies bugfix

Tuner ID detection at power-on, plus a menu extension to make use of it for AC/DC override.

-ml

>By the way, we ought to thank you really for all the work you're
>putting in to help us debug our own code and fix what's probably
>the worst outstanding bug in the 2.0 betas. So, thanks!

Thanks, Peter. But what I really want is a serial port command to "append FID" and another to "insert FID", similar to the existing "replace FID" command (#xxxx). So we would have:

   #xxxx     == replace currently playing FID (as today)

   #xxxx-    == insert FID to current running playlist

   #xxxx+    == append FID to current running playlist

If this ends up solving the lockups, I think that would be a fair trade.

Cheers

-ml

Upcoming features (sometime in March'2000):

• ir_translate capability for Sony Stalk (this one's been simmering for a while, as I'm still not 100% sure how to present it, and have been distracted by the barbarians pounding on the gates).
• The jiffies bugfix
• Tuner ID detection at power-on, plus a menu extension to make use of it for AC/DC override.
  

[nag]

• mmap() buffer for Hijack display writes
  

[/nag]

Hell yeah... I think that's a wonderful trade.

If this ends up solving the lockups, I think that would be a fair trade.

LOL

ok, it wasn't exactly difficult to implement. It'll be in the next beta.

Peter

Suh WHEET.

Also, does it just seem more common now, with about triple the number of player owners out there and posting to this BBS, happily coinciding with the v2 betas, or did it REALLY never occur before?

I don't know about other people, but I don't remember any lockups occuring on my player with 1.0x, but with 2.0x I was pretty much guranteed a lockup once a day. The difference really was that noticable.

Good. Then we need YOU to run Hijack v223 immediately. Let us know if you still get lockouts.

Hugo has rightly pointed out that this fix doesn't "make sense" in some cases, but perhaps another assumption is wrong somewhere.

Or maybe it won't help at all. Let's find out.

-ml

Already have done, I was sitting there hitting "F5" waiting for it to appear. No lockups yet...

Was this on your Mk1, or your Mk2 ?

-ml

My Mk2a, I have not used my Mk1 much since 2.0xbeta appeared, not enough to see any lockups.

FIQs have always been used on the mk1 for IR (well, definitely since v1).

On the mk2, FIQs are only used when sampling the audio inputs - eg, in aux or tuner mode, to do the visuals. In MP3 mode there is no sampling active and so there should be no fiqs generated.

Hugo

I've also only had lockups since the 2.0 betas. I'm installing Hijack v223 right now. I'll let you know if this cures it.

Edit: btw I get several lockups per day.

OK. I've just got a "stuck jiffies" error come up and the player seems to have frozen. The number after "stuck jiffies" is incrementing. It's still playing music and the hd is being accessed ok but no response from the buttons or the remote. Hope this helps.

Update: it now says "unstuck jiffies" and the counter has stopped at 44602. I can see the visual running in the background now and the player responds to the buttons and most of the remote buttons. Strangely the "Previous" and "Next" track buttons on the remote aren't responding. I've just rebooted (pulled the plug) and the "Previous" and "Next" track buttons on the remote are now working again.

Mark, I'm running 221 that you released last night. Unfortunately, the one day where I want the player to lock up, it hasn't.

More background in case it's been lost in the shuffle of other threads. Player is an MK2a with b11 and two 30GB drives (5950 tunes). I've been running fairly recent versions of Hijack. Player runs in the car randomizing the entire playlist with the least recently played mode. Always running Info:Track display. No tuner connected. Volume locked at 0db. No EQ or other DSP settings. I skip tracks quite a bit in the mornings and evenings during a 1 hour drive each way. So I tend to notice when the player has locked.

The problem has, without question, beein happening more regularly with b11 than it ever did in the past. It also happened with b7, but not with this frequency.

I'm not sure what level of fixes you've decided top put or leave in the latest version, but if you make sure to leave in whatever debug info/display you think will help you, I'll make sure to note it down when it finally locks.

If you do get a permanent fix, you might consider a slightly more substantial trade request from the empeg team... I'm sure you can vertical think of something vertical you might want to vertical aditionally ask for.

Bruno

If you do get a permanent fix, you might consider a slightly more substantial trade request from the empeg team... I'm sure you can vertical think of something vertical you might want to vertical aditionally ask for.

Not a chance.

Rob

As long as soup views are vertical, I'm cool. And I really hope you can drill down through soup entries in Artist and Album (PLEASE! )

Bruno

Stuck Jiffies 44596, about 20 playing hours after installing Hijack (approx 40% in car).

Took about 18-20 minutes before unsticking.

Mk2a, 2.0b11, Hijack v222.

>Stuck Jiffies 44596, about 20 playing hours after installing Hijack
> (approx 40% in car).

>Took about 18-20 minutes before unsticking.

Very useful information. The 18-20 minutes is actually 19 minutes and 53 seconds, the time required for the jiffies counter to wrap around a full 32-bit rotation at 3.6Mhz.

V225 should fix that problem (the bug is not Hijack's, but the fix is). Please report back if you ever see it again while running Hijack >= v225.

Thanks