That's a very plausible headcanon. Knowing Wario, he would be too lazy keep even the outside of his castle tidy when it gets dirty over time. Also, think about how much rent and heating costs a castle will bring. I think Wario loves castles to live in, but he's probably not very interested in permanently inhabiting one.
Wario just buys new living spaces because he's that rich. Either that or when he trashes his old place from doing one of his great schemes, or just from being unwilling to tidy up, he doesn't bother cleaning and buys a new place instead.