Here's an update for those with an interest in this on-going issue.

We shipped back a clients player which, we understand, suffers regularly from the sticking buttons problem. After extensive button-pressing in circumstances similar (maybe even identical) to those described by the client we have been unable to reproduce the problem.

We're now trying to determine what could be different about the client's environment/actions to cause the problem to manifest itself. It seems likely that if we can reproduce the problem reliably, we'll be able to fix it, or at least find a work-around. It's perplexing that we haven't been able to reproduce it even once, whereas the client can consistently experience it within the space of a few minutes.

We'll keep trying!

Rob

Very perplexing! How about "shipping" one of your clients back to your HQ as a consultant to demo the procedure for sticking a button?

I'm waiting for a Mk2, but I am one of those with a system that can become very very loud, so I'm eagerly awaiting resolution on this bug, because I don't want to go over a cliff unexpectedly one day.

Calvin

I'm eagerly awaiting resolution on this bug, because I don't want to go over a cliff unexpectedly one day.

Not to worry, Eternalsun. The Mark 2's will have a knob for volume instead of buttons. In theory, the knob won't suffer from that problem.

It's still in Empeg's best interests to get to the bottom of the problem, though. Whether it's a software issue, a PIC issue, or a physical problem with the buttons, non-volume manifestations of the bug could still be troublesome for Mark 2 owners.

Tony Fabris
Empeg #144

A thought on that. Another user have posted he didn't see the problem - unless he mounted it in the car, then the problem have started.

Maybe it's happening only in the car environment, meaning, dirty power source, interferences, etc.

To exlude dirty power as root for the problem, I would suggest to leave the empeg in the car but feed it from a seperate battery. I will try to do that - if I find the time.

Jo

> Maybe it's happening only in the car environment

The client that loaned his player to us has been using it exclusively at home.

We even started thinking about the frequency of the background lighting affecting the IR in some way, but this client is from Western Europe so he's on 50Hz the same as the UK.

Rob

Hopefully it won't come to bringing a client in - I'm starting to suspect that it has to do with some environmental conditions but I can't work out what. It's not temperature because clients are experiencing the problem in a whole range of climates. It's not voltage related because it happens both in-car and at home, at a variety of AC standards.

I guess we must be missing something obvious - or, rather, something non-obvious!

Rob

I'll try playing with mine more inside to see if I can get it to happen outside the car with mine. I use the buttons on the front for everything now, since my remote disappeared. The only thing I saw different in my usage patterns is that in car, I adjust the volume the most, while inside, I use the buttons for navagiting the menus to play songs.

I will also be adjusting the amp, as 0db was a bit loud. I found this out by suprise from a bug I created. I have mine play a startup sound on boot, and disabled it for the installers. I then got my car back with the newly installed empeg, hooked up a serial cable, and restored my sound. One thing I never noticed is that the sound player I compiled plays back at 0db. It was a rather scarry moment to be parked testing it out, and hear the startup sound that loud. After that, the empeg has stayed out of the dock until I got back into it at home with no audio.

My empeg site is:http://24.236.3.131/empeg/

This is really developing into a nice little mystery :) I am very curious about the solution....

This reminds me of the problem I had on my Twingo: two years ago ABS controller would decide something was wrong and switched itself off whenever I was returning from a consulting stint in another city (it was always Friday, with the car being parked for a week). Low battery, moisture and similar were blaimed, but the culprit was not found before intense competition in cellular phone market caused GSM antennas to sprout on every other light pole in my city, and ABS started missbehaving here, too. It turned out that, due to faulty shielding, my ABS controller was picking spurious signals wrom one of wheel sensors near strong radio sources, and there was a local radiostation near which I was passing when returning from that other city.

I feel the solution to the sticking button will be something to put in a book on complex device debugging...

Cheers! (and good luck)


Dragi "Bonzi" Raos
Zagreb, Croatia
#5196

This is the first time I've written anything about this 'sticky keys' matter.

First I just want to say that I never experienced any problem with 'sticky keys'. And I use my Empeg alot.

Here is my environment.

- WV Caravelle (Eurovan) Diesel Engine.
- Wired directly to the battery. (With some fuses, yes....)
- Norway, Autumn /Winter / spring -20 / +20 centigrades.
- Evening, daytime light.
- I'm using a Kenwood steering wheel remote, as well as the supplied remote controll.
- Serial # 302. (Any HW revs. out there??)


These are some thoughts I had, don't know if they help.
Maybe one should compile a 'database' with some parameters like this and just
start to rule out whats common/not common??? (Another thought.)

Regards
TommyE

I don't see why the knob would behave any differently. I imagine it will still operate via the same PIC, and rotating the knob will send a stream of pulses either a lot of volume ups or a lot of volume downs, or whatever.

Calvin

Yeah, but that's only if the problem is in the PIC. There's no proof that the problem lies there. In fact, Hugo's pretty sure that it isn't there.

Tony Fabris
Empeg #144

No, this is different totally.

With a button, a button down code is sent and a button up code is sent. The player software does the autorepeat if it gets a down and not an up. The bug appears to be that the main CPU doesn't see the button up and so keeps repeating.

The rotary control sends one code for clockwise & one for anticlockwise. If the CPU misses either of these, the worst that happens is that the one click goes missing - there's no runaway condition. The Mk2 buttons also send two up codes on every button release to try to ensure that at least one gets caught. We're still investigating the missing of buttons, using a patched kernel to look at times when IRQs are disabled (which can cause a button to get missed).

Hugo

Hey guys,

Could we try an experiment with the IR?
Can people who are expiriencing regular volume sticking problems TAPE OVER the IR receiver on the front panel. This would help identify whether the problem is purely an internal one.
Maybe the IR is getting a random burst during a key press. in the confusion of decoding what it got, it 'forgets' to send the key up.

I don't get this often enough to help in the test (twice in six months are not good odds).

____________________________
Murray

I don't know if the sticking button condition could be related to the IR. As I understand the way IR signals work, a runaway condition couldn't happen because of them. IR signals don't send button down and up codes as two separate things. If they did, you'd induce runaway conditions all the time simply by moving the remote before releasing the button. The way that IR signals get a button to repeat is that there is a "repeat last command" code in the IR commandset. If you hold down a button, it sends the code for that button followed by stream of "repeat" codes.

Now, your contention is that the runaway condition isn't caused by IR commands, but that the system might be getting confused when it receives a random IR signal combined with a button press. This is a good guess, but if that were the case, wouldn't we be getting spurious random commands activating the unit even when we weren't pressing the buttons? For example, the unit is designed to wake up from sleep mode at any button/IR signal. I've never seen it wake up without my explicit instructions, have you?

Besides, I spent a lot of time trying to induce the bug by deliberately combining IR commands and button presses. The frequency of the ocurrence of the bug was no greater than with button presses alone.

Tony Fabris
Empeg #144

OK, here's the current situation.

We've reproduced the problem on the specific player three times. Unfortunately, all three times have been when there wasn't much debugging code in the kernel to fathom out what broke :-(

We have reproduced it without any notable stray IR so this is not (the only) cause of the problem.

My current belief is that the failure has something to do with the unit being left powered but switched "off" via the menu or top button for a while. I have reproduced it within a couple of minutes when the unit has been left powered overnight but never right after a hard reboot. This makes investigating the problem a slow process since debugging generally involves putting a new kernel on which requires a power cycle :-(

If anyone feels like trying top reproduce it I suggest they try doing so immediately after a cold boot and report results.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

Wow, this is really cool information. I'm glad to know you guys are working so dilligently on this. I'll try reproducing it today in the way that you described. If I can make it happen, I'll let you know.

Tony Fabris
Empeg #144

Having reproduced the problem we think we've fathomed out why it was happening. Hugo and I spent quite a bit of the Easter bank holidays rewriting parts of the IR driver. You can find an interim upgrade that incorporates our changes at http://www2.empeg.com/upgrades/test/interim/, please try it and let us know if it solves the problem.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

Mike, would you be willing to briefly lay out what was happening? I confess I have developed an unhealthy interest in this subject.

Mike, would you be willing to briefly lay out what was happening? I confess I have developed an unhealthy interest in this subject.

It basically looked like _very occasionally_ an interrupt would be missed. The IR driver was already detecting this but it was difficult to see the information when in-car (it's in /proc/empeg_ir). The driver knows that an odd number of interrupts have been missed since it is edge triggered on either edge and therefore the level should alternate between interrupts.

We've solved the problem by producing a simplified interrupt handler which just collects timings and analysing them later in a bottom half. We then recoded the interrupt handler as a FIQ[1] routine which has very low latency. The standard Linux kernel never disables FIQs (and we only do to avoid concurrency problems within the IR driver itself).

[1] FIQs are fast IRQs. The handler starts execution immediately without going through a vector and has a banked set of registers so they can work without saving the state. They must be coded in assembler to be useful.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

This sounds absolutely fascinating.

Was the change in the player software only, or were there changes to the Kernel? I only ask because I want to know if I can continue to use my re-compiled 10a kernel with my custom bootup image with this interim file.

Tony Fabris
Empeg #144

The changes were in kernel only. The patch is on the website linked from the interim download page.

Hugo