For some people freedom means being able to violently deny others to roam the land, to deny access lakes, to deny foraging for berries and mushrooms and even to deny them access to the coastline and the ocean.
For them, freedom might entail being able to deny these basic modes of being e.g. based on monetary worth, social standing or even ethnicity (like country clubs in the US).
If that's what you consider freedom, I don't think I'm going to be able to convince you otherwise.
I already explained what I meant by damage. Being unable to see how the freedom of a tiny minority shouldn't trump the freedom of basic modes of being of the vast majority, that's damage.