Vance says he’s annoyed he won’t have the chance to debate Harris