But the majority of us don't want to leave the UK