Thinking about it, I understand conceptually why they do it that way. When in colorized mode, each character is merely a pixel, and the character color governs the value. When in ASCII mode, each character is a brightness value. There's no way to combine both without deliberately throwing away the usefulness of assigning each pixel a color.