What would happen if I keep obstacles as they are?

Keeping obstacles as they is not good for your base.
With the obstacles on the way, you cannot place your buildings and walls where you want them to be which alters the layout of the base that you originally want.
Taking obstacles regularly from your village is important especially when you progress to higher town halls because new building will be made available and you need to place them in order to improve your village’s offense and defense.
In removing obstacles, you will need a builder, and so take note of when your builders will be free from building or upgrading something.
When a builder is free, remove all the obstacles, or as much as you can if you have limited resources to do so before making them build or upgrade another building.