@5357ziv There was a post here with a reply from f.lux that they need to get an HDR monitor to test out issues there. And that a user said they disabled the HDR setting in Windows 10 and it seemed to hide the issue for them.
I'd guess your built in laptop screen is standard color and brightness, and the 5k screen is wide gamut and HDR.
One question I'd have...is SnagIt supposed to get exactly what you see or what would be shown normally without f.lux?
It'd be nice to get the normal UI, but I can imagine it not knowing how to remove whatever f.lux does to the screen. I have no idea if there is an API to allow changing what users physically see without altering what other programs read (for screenshot grabs).