My wife and I have done this once (in '23, our first trip to BVI) and I think it's a fine strategy. Keep in mind that the ferry skips a time slot a couple times a day (for meals or shift change or something) so you'll want to plan around those gaps.
As for a taxi, we did not prearrange one and we had some issues. This was due to cell phone problems, mostly. In the end, someone was nice enough to help us call one, but I think next time we'll try to prearrange just to be safe.
.
