Yeah, the big issue with saying “it’s your land do what you want” is that you start getting wildly-unsafe buildings and situations because “it’s my land I can do whatever” rapidly turns into “and I want to build something highly unsafe to live in because it’s cheap to do so”. This is how you wind up with tenement housing where one apartment catches fire and a hundred people burn to death because nobody forced the builder to consider evacuation routes.