Just about all of these modems/routers that do QoS are doing it with a Linux kernel inside. The QoS / Traffic-Shaping is a stock kernel feature on Linux, so look around at the available documentation on how to use it.

Eg.
http://lartc.org/
http://lartc.org/howto/lartc.qdisc.html

The knowledge gained there can be applied to Tomato firmware setup, or used on a dedicated Linux box (any old/new hardware will do) that you could install to manage the bandwidth and routing.

Cheers