This is indeed correct, I'm just using HSV instead of YUV, but the primary source of information is the luma/brightness component (although currently all 3 of the HSV components are averaged, so perhaps a better weighting may improve precision).