Successful applications can come in many forms. You should approach the organization application like you would a resume: this is the avenue through which you convince Google's program administrators that you are qualified for the job of mentoring students.
The most important part of GSoC is providing the students an excellent experience over the summer, and Google chooses organizations that they feel confident can do this based on their application.
The Ideas Page
The ideas page is the most important part of the organization application. Please see the Ideas Page chapter of this manual for more information.
One question Google consistently asks of an organization on the application is whether they have participated in previous years, and if so, what their students' pass/fail ratio was. While pass/fail ratio is but one indicator of the success of the previous year, it does weigh against other components of the application.
New Orgs vs. Returning Orgs
Every year Google tries to make room for new organizations in the open source world who can provide a different perspective or different opportunities to the students who participate in GSoC. This can sometimes mean rejecting an org that has successfully participated in years past to allow space for new organizations. Often these decisions are very difficult for us to make, because they don't have much to do with the returning orgs' success in previous years.
What Not to Do
Incomplete applications are common and hard for us to review. Not answering questions on the form, not taking time with the quality of your Ideas Page, and not executing well on your application can all be avenues to rejection. For example, don't submit an ideas page in the form of a Google Wave our administrators don't have access to. Google doesn't have the time or capacity to iterate with you on your application, unfortunately. Take the time to create a thoughtful proposal.
Just Because You Did Everything Right...
...doesn't mean you'll be accepted. Every year Google receives many more applications for organizations that want to participate in GSoC than it has capacity to accommodate. Some of the decisions about which organizations are accepted and which aren't come down entirely to space available in the program. Every year, Google offers an opportunity for rejected orgs to receive feedback via email or IRC on what they could improve in their application for next year. Sometimes the feedback is specific to the application, but sometimes the feedback is simply: "We can't accommodate everyone; please try again next year."
A Note on Umbrella Orgs
Google's program administrators actually look quite fondly on the umbrella organizations that participate each year. It serves a dual purpose: it allows Google to accept more organizations in the "space" of just one, and also gives an opportunity to accept a marginally-topical org by putting it under the umbrella of a related org.
If your application is rejected, the following you may want to consider reaching out to an accepted umbrella org that might be able to accommodate you.