On a Mac, Quicktime is the OS level media framework. Anything Apple related will use it (like iDVD), so previewing a file first in Quicktime is a good thing to do. It can be extended with new codecs, just like the Windows Media layer on Windows.

VLC doesn't use Quicktime, instead it has it's own codecs built in.

The easiest codec library I've found on OS X is Perian. Once you install that, Quicktime should be able to handle about anything VLC does, as both projects pull from ffmpeg.

*edit* also somewhat handy, in Quicktime, Command-I will open info on the file, may help narrow down what codec the audio side is using.