NSVisualEffectView with mask has jagged corners
I was trying to achieve this with a NSWindow view a while back and I came across this awesome library: https://www.cocoacontrols.com/controls/waythedarkside
It allows you to add a light or dark blur to your view/background. I think this is what you are looking for :)