Exactly, why should the OP have to do anything because of a stupid idiot?It sounds like you want to keep your land, and rightfully so. If so, then make him remove the driveway. That's not being an A hole.
OP, stick to your guns and make him remove it and keep your land. The idiot is going to hate you no matter what, unless you give him the land.