The standard calculation of luminance is something like Y = (0.299*r + 0.587*g + 0.114*b). At least, that's what I've used in my own code in the past. You can do the calculation in integer arithmetic as Y = ((299*r + 587*g + 114*b) / 1000).

Perhaps for the selected node, we could simply invert foreground and background if it's a coloured or modified node? And use the normal highlight foreground and background (textHighlightText and textHighlight) otherwise?