If the neighbor has been using that bit to access his property for seven years (in Georgia as mentioned above) and he's not using the bit for any other purpose, he has a right to continue using it. If it's been less than 7 years and no pre-existing easement was granted by the previous owner of your property, I'd try to work something out with the neighbor or just not worry about it unless he goes to doing something else on the bit he uses.