One of the things I've found to be trickiest about designing a game is expanding in all the necessary ways from the initial idea. It's nice to come up with something like "let's make a treasure hunter guy like Indiana Jones and put him in all sorts of historical settings like Egyptian ruins!" That certainly sounds like a solid idea (forgive me for my bias, this is the game I'm currently working on...) but it's a far cry from being specific enough to actually build a game on.
There's a lot more to it than that, including but definitely not limited to things like what enemies/conflict will there be, how big will the game area be, whether or not there will be non-hostile characters for the player to interact with, and so on.
I've been working on nailing down more specifics to get the game more fleshed out, and when I do this it's easy to take things too far. For instance, in a side-scrolling action/adventure game, to get to higher platforms or across wider gaps, double jump is a pretty standard game mechanic, but what if I think about adding areas that are a little more out of reach than that, the player would need an ability that can allow him to get to those areas...should the player be able to fly with a jetpack or something? It's too early to tell on things like this, and that's the point where I have to step back and say, "I'll come back to this later."
Flexibility is important in a game's design, I think, but changing and adding design elements should be considered carefully. I remember reading something that went something like, in contrast to thinking outside the box, it's better to think on the edge of the box. To put it in my own words, it's good to expand your thinking, but stay in familiar territory or things can quickly get out of hand. To the above example of player movement abilities, giving the player a jetpack may not quite be the perfect idea, but it's probably a lot closer than, say, having the player turn into a bird to be able to fly. Both examples here accomplish the same goal, but one would feel far more abstract in the world I'm trying to build.