I'd describe it as a lot colder (not "a bit cooler") than the Caribbean islands in the winter. It's roughly at the same latitude as North Carolina, so that's a pretty fair comparison. Often people fail to realize that it's actually an Atlantic island, not Caribbean. So think of it as beach time there equivalent to beach time in most of the States.
Just so long as you don't go expecting it to be a beach vacation, it's a charming and beautiful island and can be appreciated even without lying on the beach. We went for our honeymoon a gazillion years ago, over the Memorial Day weekend, and it was marginally cool for lying on the beach.