WL4 marked a changing of the guard for the Nintendo R&D1 division. It's safe to assume that the old Wario staff and stalwarts were calling it a day. They had created five unique Wario Land titles by that point (don't forget about Virtual Boy Wario Land), and had likely done everything they set out to do with the franchise.
They were never the sort of studio to churn out sequels without a good reason. They were very creative thinkers, and you'd never see them pulling a Mario, Pokemon or Megaman, and just pumping out more of the same old thing. That wasn't their style or motivation.
That's why each Wario Land title stands on its own, compared with Mario's 2D outings that are more or less the same (same worlds, enemies, scenarios, gameplay, music, etc.)
It makes perfect sense that R&D1 would want to end the franchise on a high note, rather than keep on beating the same horse. Wario Land 4 tries to wrap the series up in a polished package by streamlining the treasure hunting, revising the gameplay and covering all bases, whilst adding new twists and themes.
It's important to note that during the game's production the staff was a mixture of old and new. The most notable new staff member was Goro Abe, who became the chief director of the WarioWare franchise later on. Even though WarioWare didn't exist yet, you can definitely feel that style and influence in WL4 (the baseball minigame speaks for itself).
The other key reason Wario Land ended was the fact that Nintendo R&D1 was shutdown in 2004 and its members scattered across the company.
Wario became homeless after this and has drifted about in limbo ever since (likewise with Metroid). For a few years he had the WarioWare team to fall back on, but apart from that, the classic/general version of Wario has only received two games since the 2004 shutdown (Master of Disguise and Shake Dimension). Both of these were made by third parties and neither of them were upheld or promoted in things like the Super Smash Bros or Mario Kart franchises, despite coming out in the same time period.
We're very lucky that Shake Dimension was made at all! The only reason it exists is due to one old member of Nintendo R&D1 (Takahiro Harada) wanting to make a new game in the old Wario style, and bring about a return of his classic portrayal. So he took his idea to the newly-formed company GoodFeel and Wario Land: The Shake Dimension became their first game for Nintendo.
Shake Dimension didn't revive Wario's classic legacy, mind you. That honour belonged (and ended) with Wario World, which marked the ten year anniversary of the character in 2003 and the final game in his original legacy (the final appearance of his castle and immense riches). It came out just as the WarioWare series debuted, marking the end of Wario's classic era.
Nintendo R&D1 wanted to make a 3D installment of their Wario franchise, so they approached their old friends at Treasure to help them make it.
It's sad to think that the year after Wario World released, R&D1, Nintendo's oldest studio and most historic division, was closed down.
It's also sad to acknowledge that Wario World marked the very last time Wario had a happy ending in a game. I suppose Nintendo reviewed the character sometime after Wario World and decided they didn't want a selfish, rotten crook getting everything he ever desired, as he did in Wario World with his castle full of riches (or every Wario Land game before it). Giving a bad guy a good ending probably didn't mesh with the mainstream family-friendly image of the Mario games, so Nintendo turned him into a permanent loser, always looking to get-rich-quick. We never saw his castle or wealth again. He is always shown living in poverty in a dingy house from that point on, with no wealth or happy endings for him in Master of Disguise or Shake Dimension.
Regardless of how Nintendo feel about Wario, he can always be happy that between his inception (1993) and WL4 (2001) he was the only Mario-related character to receive adventure games for Nintendo's portables (Rare's Donkey Kong games aren't really related).
It's amazing and funny to think that after Super Mario Land 2, Mario didn't receive a new handheld adventure until... New Super Mario Bros. on the DS, which was years after Wario Land 4.
The reason for this is because Nintendo R&D1 created the Game Boy systems and were more comfortable creating games with their own characters, rather than making new Mario adventures for their Game Boy systems. That's why they stopped making Mario Land games and created Wario Land to solve this problem.