From my travels pre covid:
Antigua: Jolly Harbour and English Harbour get my vote. Both are small communities, one is planned with boat docks for a back yard and the other is boat and pple watching at Nelson's Dockyard.
Grenada: Grand Anse area for the culture and spice smells once you head away from town. Food was good too. Less touristy.
And wow the bvi!!!
St Lucia is an option as well. I found the water rough in Rodney Bay so cannot comment on this.
Also, bequia in St Vincent though medical is a short ferry away. The Grenadines sparkling blue water is mesmerizing.
Confused yet? Try them all!