The Speculist posts on push prizes and getting to the Moon. It's in favor of relatively modest (in aerospace terms) space prizes like the Centennial Challenges, but suggests that the boundary where prizes are too ambitious is somewhere around the difficulty of the Lunar X PRIZE because there's a threshold of difficulty where challenges are too difficult for small teams, and large corporations with stockholder requirements can't risk going after the prizes.
I think there's probably a great deal of truth to this line of thinking. However, an extremely difficult prize can be designed to reduce this effect, for example by increasing the prize reward, reducing the risk by including second and third place prizes, including incremental sub-prizes that are stepping stones towards the big goal, and/or orienting the prize towards a goal that businesses would find attractive to address (just not quite enough to go after it without the prize) because of market potential. Another way of thinking about it is that there's no harm in offering a large prize even if you as the sponsor think there's only a 10% chance of anyone achieving the goal, since you don't have to pay if noone wins, and some of the competitors may make useful progress towards the goal even if they don't win.
The post links to a number of earlier posts from the same site on space prizes, such as this one that's only a day old about the organization in the space science community for a change in the Vision for Space Exploration to focus on astronomical satellite servicing at Lagrange points and exploring asteroids instead of the Moon. In that post, prizes are seen as a third alternative.