Think about “How to feed that many people/animals”, and how to get rid of the wastes, and you’ve done on a long-way in your castle design. It’s also more interesting because suddenly, you don’t have a linear dungeon with a door and an exit, but tons of different way to access it.

I feel like I remember Matt Mercer saying something similar to this when talking about creating worlds, and then Brandon Sanderson said the same thing about his books, so I watched a few of his lectures to help with my world building.

A lot of what he does to make his worlds feel alive and real and expansive comes from putting himself in that world and thinking about what it would need to actually operate, then answer those questions with what's available in that world. How do they feed themselves? Get water? Dispose of waste? Settle disputes? Travel when it's too far to walk? Then you expand on those to answer what those things need.

They ride horses? Gonna need a stable and grooms and tack. Where do the grooms live? Who's their boss and where does he go when not bossing around the grooms? Where does the tack come from? Made in house, gonna need a leatherworker and their own shop and place to live. Horses are also gonna need hay and to store it. Now you've got a farm and storage. A farm? Gonna need farmers. Where do they live? OP mentioned a wizard, so they have magic. Would they have used magic to make any of these jobs better/easier? Maybe the Wizard is upset about having to use his gifts to assist with lowly jobs and there's some old note of him complaining? Now your players know to look for a wizard's quarters from checking out the stables.

All of that just from them having horses. A little bit of follow-through can turn a boring map into a living world. It's great.