How should I handle this?

I think you should call him. He probably just thought you were ignoring him, because you kind of blew him off. So he would be happy if you gave him a call. You have nothing to loose, if he doesn't want to talk to you it least you tried but, i think he will be very happy you called. Hope i helped.