I suggest, you might use a expression like "gentleman caller," but only if you do not mind sounding such as you're a personality in Jane Austen novel and only if you don't intellect that, though it does eliminate the infantilizing "boy" misnomer, it falls short of denoting The person is elderly, under no circumstances mind how it may be a misnomer