I've weathered some big north swells in Diamond Cay and it was always quite settled, tucked between Foxy's Taboo and the B-Line bar. The holding isn't great, but there are lots of mooring balls.
Here's an image of the waves on the north side of Diamond Cay while I was anchored in calm waters just a few yards away to the south:
The only other truly calm anchorage during a big north swell that I've found in the BVI was the North Sound on Virgin Gorda.