That's a very plausible headcanon. Knowing Wario, he would be too lazy keep even the outside of his castle tidy when it gets dirty over time. Also, think about how much rent and heating costs a castle will bring. I think Wario loves castles to live in, but he's probably not very interested in permanently inhabiting one.A big house is too much trouble to keep clean and Wario wouldn't pay anyone to clean it. Hence why he downgraded from a castle.