Have you considered going to the police? The idea that he has your actual phone number is pretty scary. He has to be somebody who knows you. I realize he hasn't committed a crime, but getting a report on the record could be important down the road. Since you have his phone number, that could be an important bit of identifying information if (in the unlikely event) he starts stalking or further harassing you.

Reporting his phone number now might make getting an effective police response in the future a bit more likely.