Oh, I'm all over that, Dave! The mansion they film at is in Cold Spring Harbor. It used to be a military academy that I played basketball at once a week in the ballroom. My golf partner got married there 20 years ago and we remarked that they took down the baskets.
Even this show pans a lot of CA lhs, I guess the 26 on LI are not enough! The show uses many street scenes in Hamptons but I think the majority is filmed in CA.