If conditions were right I'd do it. That will be almost a mile, in the dark and you have to get around cistern point. Be sure to bring a flashlight so you can be seen! Also, make sure the company is ok with anchoring in Haulover - some may consider it a day anchorage only but I have read there is good holding.

Life involves risks, take some prudent ones (NOT with the BVI ferries)!