Walls take no building time and instead are instant upgraded so it seems odd to need a builder available. Not requiring a builder would allow the builders to be use for long projects and walls can be upgraded on demand. 5 days just to get a builder available to upgrade a wall segment seem ridiculous so lets just make walls builder free to upgrade
I know this is frustrating. It's been frustrating to me as well. But it's part of these types of games. They are designed to make you want to spend money. Plus, if everyone could dump their gold whenever they wanted, they would have less available to loot.
