A very interesting thing happened in the last week - I picked up a copy of Starcraft, and with it a realization about the progression of RTS as a genre. Firstly, though, I feel obligated to defend myself for the terrible oversight of not having played Starcraft earlier. The reason I didn't pick up Starcraft when it came out was simple - it looked like Warcraft in space, and I'd played Warcraft, so Starcraft seemed redundant. In retrospect that was a very silly thing to think, but that's my excuse.
Anyway, I have since played just about every other RTS game I could get my hands on, so the experience of playing Starcraft has effectively been a way to return to what you might call the "core" of RTS games and understand the overall trajectory they've taken. What struck me immediately about Starcraft as someone who hadn't played it before is that it actually felt "fresh", moreso than some more recent RTS games, even though it's over a decade old. The races are more differentiated and well defined than in most other RTS games, and yet it's clear that every unit has been meticulously balanced. The gameplay is aggressive, chaotic, and micro-management intensive, and yet immediately more comprehensible than in many games that are less chaotic.
That's enough gushing about a game that's already got plenty of acclaim. The point is that I was expecting to enjoy Starcraft when I bought it, but I wasn't expecting to think that it was still better than most games in its genre made in the last decade.
So what's the difference? It's actual quite simple: what Starcraft has is fluidity. The game isn't about building up a huge base and teching up - it makes you branch out, build forward bases, and constantly skirmish with your opponents. There are a lot of small design decisions that work together to make this work, but there are a few that stand out as missing from more recent games (yes, it's time for a numbered list!):
1) Base defenses are weak. That's not to say that they're not useful, but they're ultimately there for support - you can't just build a couple photon cannons and then consider an area safe. This is even more true for the Terrans and Zerg, who lack a single anti-ground/anti-air defensive structure. All defensive structures are available very early on, and useful for repelling rushes, but by the end of the game they aren't really turning the tide of battle anymore.
Compare that to more recent games: AoE2 and AoE3 both have upgradeable towers AND castles (forts in AoE3) that can only be overcome by full armies. In Command & Conquer Generals, one general can build EMP missile defenses that disable any vehicle they hit, making a direct assault suicidal. Perhaps the most egregious case is Supreme Commander, in which you can cover your defensive turrets with energy shields to make an impregnable fortress.
Building powerful defensive structures is so popular that it's become its own genre of game (Tower Defense). While there's certainly a place for games that are about building towers that shoot things, effective base defenses have become an assumed in RTS games and I'm not sure most designers recognize the kind of effect it has on gameplay (hint: it's called turtling).
2) Resources are not infinitely reproducible. As far as I'm aware, it was Age of Empires that introduced farms and the idea of renewable resource gathering. Like effective base defenses, it has come to be an assumed part of many modern RTS games. This, more than anything else, enables turtling, because expanding your territory is always risky, if only because you have more ground to defend, whereas building more resource-producing buildings comes with no risk at all.
Getting past turtling economics in RTS games doesn't strictly require that resources in your base run out, however. It can also be effective to simply limit the rate at which resources can be gathered from a single base. In other words, the important thing is that you can't endlessly grow your economy without opening yourself up to risks.
3) Maps are full of chokepoints. This feature has more to do with level design than systems design, but it's clearly part of the overall gameplay concept. In most RTS games, you're building on open plains with occasional geographical features of interest, or else on islands connected by water. In virtually every map in Starcraft, your base is in a fairly small, defined area with 1-3 points of entry.
The significance of this is that it's what makes turret-less defense of your base possible. If you know that your opponent is going to come from one direction, you can concentrate your forces there and stand a good chance at repelling attacks. On open plains, no matter where you place your units, the enemy, if they scout ahead, will be able to go in a small circle around them and enter your base. To avoid this, you need walls or other defensive structures you can place around your perimeter to buy you time to respond to attacks...which leads us back to point 1.
So overall, it seems that in the last decade RTS games have become more about building (and thus about defense) than about fighting. That's not necessarily a bad thing (building is fun!), but the fact that I found Starcraft to feel "fresh" reinforces my belief that there are very few more recent representatives of the Starcraft model of tactical-skirmish-centric gameplay. I believe that Dawn of War II sees itself as being such a representative, but I find it kind of hard to get into the game for several reasons (forced Windows Live registration plus a CD-key even when you buy off Steam, it's laggy on a computer that meets "recommended specs", there's no real tutorial, etc.).
I'd like to see more RTS games taking some of these points into account.