Does the icon stay on all the time with the key in that position? Or just briefly flash?

If the former, yes I'd worry because it means the car's electrical system isn't putting out 12 volts unless the engine is on.

For me, I only get the battery icon when the engine's off if the player hits a deep bass note with the volume turned up (that's when the subwoofer amp draws the greatest amount of current).