My guess would be that the battery isn't providing a constant voltage, maybe due to a varying load on the battery as the speaker moves. That is, some sort of feedback loop. Obviously the battery never causes the speaker to move on both sides of the equilibrium point, but it might cause it to oscillate on the positive side (or the negative side if the polarity is reversed).