I recently finished Mega Man 9, and overall the game was fantastic. It also brings me back to the issue of difficulty in games. Mega Man 9 is an extremely difficult game, but in general I felt less frustrated playing it than I have playing many less challenging games, and I'm going to consider here why. First, as a disclaimer, I recognize that I'm the kind of player who enjoys serious challenges, and that not everyone will - players who don't like to die more than a couple times on the same level will not enjoy Mega Man 9. That said, for those players who are looking for a challenge, Mega Man 9 does it right.
The hard part of balancing difficulty is figuring out the difference between a challenge and a frustration. I'll illustrate with a couple examples. It's a challenge when you're handed a portal gun, and you see that you're supposed to use it to get to the other side of a giant pit, but need to figure out how; it's a frustration when your high-level wizard is forced to crawl through a dungeon without being able to cast any spells, and risks getting killed by the kind of creature that they could normally off with one fireball (I'm looking at you, Neverwinter Nights). If you play a lot of games, then you've likely seen countless examples of both (and probably examples of both in the same game on many occasions), but it's hard to quantify the difference between a challenge that is fun and one that makes you want to throw your controller across the room. So let's attempt to quantify the difference.
The first thing we need to ask is, "what makes games fun to begin with?" Of course, this isn't an easy question to answer, and a many different ones have been composed (in some cases by the same person). I tend to agree with Danc over at Lost Garden, though, that learning is, if not the only source of fun, at least the source of fun most core the game playing experience. So why do players enjoy games that are challenging? Because if you've played many games, you've probably gotten pretty good at them, and if the game you're playing only includes elements you've already mastered, there's nothing for you to learn. If you're not learning anything new, then playing the game is just repitition (I'm looking at you, xp grinding).
So the issue is that in order to be fun, a challenge has to require you to master a new skill, but if the skill is completely unrelated to what you've been doing before, or just too much more advanced, then the player will become frustrated and stop playing. What's worse, if the "challenge" manages to decrease the odds of victory without actually introducing any new elements to learn, then frustration is the only thing it can achieve. This doesn't mean, however, that a challenge always has to come in the form of a completely revolutionary game mechanic - the truth is that even a small change (upping an enemy's hit points, or preventing them from staggering when hit) can actually make a substantial difference to the way you play the game, and thus provide a fun challenge, but this depends on how it's handled.
So enough with the generalizations, this post is about Mega Man 9. When you start playing the game, you will likely see its gameplay as "unforgiving." If you make a single misstep and fall into a pit, or land a spike (which are not in short supply), you instantly die, and are sent back to the last checkpoint. What's more, there are only 2 checkpoints in each level - one right in the middle (usually before some particularly challenging platforming) and one right before the boss, AND once you run out your lives, you have to restart the level even if you had made it to a checkpoint. What's more, the game doesn't try to make it easy for you to avoid these dangers - it's constantly throwing obstacles and tricks at you that force you to fall to your death. So even if you get past the difficult jumps, you don't get to save your progress? That doesn't sound very fun.
After you play for a while, however, you realize why this system works: reaching checkpoints isn't the primary form of progress you make in the game - learning how to traverse the level is. That is, most of the challenges in Mega Man 9, once they have been overcome, are much easier to complete a second time. This is because your victory or defeat is almost never the result of chance - when you die, it's because you made a mistake, and when you succeed, it's because you've mastered the challenge.
There are also a couple of structural elements that support this gameplay. First, the levels are actually all pretty short - the density of challenges means that it takes a long time to get through on your first attempt, but once you've mastered the level, and go back again, you realize that actual space of the level is pretty small, and you can traverse it very quickly if you know what to do. This means that saving your progress at checkpoints isn't nearly as important as it would be if getting back to that point took an extended period of time. To illustrate this principle, note that it took me 5-6 hours to beat the game, but record times are posted online, and the last time I checked, the best time in which someone had beaten the entire game was just over 24 minutes. I guarantee you that they weren't using cheats and tricks, either - that's just about how long it takes you to go through all the levels if you never have to pause and attempt something multiple times.
There's another structural element that's just as important in balancing the difficulty - you get to choose the order in which you complete the levels. After the opening story, you're taken to a stage select screen where you see all 8 master robots, and by selecting one, you're taken to their stage. There is no first stage and eighth stage - they are all balanced, but you will likely find some of the challenges easier to complete than others. If a stage is too hard for you, you can leave and try another one. Not only do you have control over the order in which you face the challenges, but as you complete challenges, you gain new abilities, which makes it easier to overcome the stages that gave you trouble in the beginning. By arranging the levels in this way, the game has effectively built in automatic difficulty balancing. Rather than having to perfectly execute everything on your first attempt, you're free to explore the various levels (mastering the basics as you do), until you find one that you think you can complete, and as you begin completing levels, you're getting stronger (and more skilled), and gradually making progress toward overall completion, not just in terms of checkpoints passed, but in terms of gathering the resources you need to overcome whichever challenges pose the most difficulty to you.
The total effect of this design is that you feel like you're constantly improving and constantly making progress even when you fail out of a level and lose your checkpoints. Combine this with the fact that many of the challenges are creative, engaging, and amusing, and you have a recipe for a great game experience. Even though the levels try to "trick" you into falling into pits and onto spikes, the game is straightforward about it - it communicates, not through text, but through visual cues and samples of gameplay elements, that it's going to try and trick you in the next room, so keep an eye out for it.
Not every game needs to be as hard as Mega Man 9, and this certainly isn't the only way to structure a game in order to be challenging and fun, but it serves as a great example of a game that uses well-crafted design to deliver to their target audience (crazy oldschool gamers) exactly what they're looking for.
Besides, who can resist a game with a boss named "Galaxy Man"?