April 20, 2006 - Bjorn Freeman-Benson, EclipseCon 2006 Conference Chair
The EclipseCon organizing committee is serious about improving the quality of EclipseCon each and every year. To help us plan for EclipseCon 2007, we surveyed the 2006 attendees - this document is my summary of the survey results. The complete survey results are available here. I encourage readers to examine the results themselves and to correct any misstatements or omissions in my summary and conclusions.
We received 338 responses (332 from attendees) for a spectacular 26% participation rate1. The mean satisfaction value of 1.87 which I interpret to mean "More than very satisfied" (1 = Excellent, 2 = Very Satisfied). We also received 31 comments via the general conference feedback form on Eclipsezilla.
Naturally, we love to quote the compliments we received (although later in this summary I will admit that not everyone was estatic). Let's start with some of the best:
Ok, enough of that :-)
Onwards to the core of the feedback...
70% of the respondents felt that the EclipseCon 2006 seven parallel activities (five talks, two demos, and an exhibit hall) was the best choice. 22% felt there should be fewer than seven with the most popular request being five parallel activities. From an organizers' point of view, five would be operationally easier but would reduce the number of opportunities for the community to speak. We will probably have around seven parallel activities again at Eclipsecon 2007.
The other conference format question was about whether EclipseCon should have in-advance tutorial days (and, if so, how many) or in-parallel tutorials. 71% of the respondents preferred the tutorials in advance format and thus we will keep that format for 2007. 53% wanted more tutorials which is a good number, so we will consider a second day of tutorials if it does not increase the cost too much (convention center rental, A/V, etc).
I proposed (in survey question 6), increasing the technical day by an hour to fit in more sessions, but 65% of the respondents said "no". Ok, good enough - we won't have a longer day.
68% of the respondents thought we had the right number of keynotes in 2006 and 27% thought we should have fewer. Hmmm. The most common answer for "how many fewer?" was three - one per day. That makes sense, especially as a keynote takes the same number of time slots as five other activities. We will probably have three keynotes (one each morning) at EclipseCon 2007.
When asked who you would like to see as keynotes, most of you (66%) didn't answer, but of those who did, the most common answer included "Joel". I wonder if some of the respondents misread the question and were answering "which keynotes were best?" rather than "who would you like to see next year?".
There were a few specific suggestions: Josh Bloch, Jeff McAffer, Tim O'Reilly (again), Robert Cringley, and a number of people I didn't know - we'll pass all the names on to next year's Program Committee.. A few people wanted Bill Gates or someone from Sun (both are rather unlikely, don't you think?). This person probably had the best advice:
The best keynotes are usually by outsiders from completely unrelated domains that solve a problem analogous to one being faced by the Eclipse community (such as openness, scalability, community building, etc) Insiders or executives from member companies just make it feel like an Eclipse love-in and don't provide much insight.
The Exhibit Hall is an essential part of EclipseCon because EclipseCon is a place where the entire Eclipse community gathers - users, contributors, committers, and commercial interests. The Exhibit Hall operates (mostly) in parallel with the technical sessions, so we asked a question about what the Hall open hours should be. There were a wide variety of suggestions from the half of the respondents who answered this question, but there seemed to be a preference that:
There are some logistical/operational issues with having the hall open for all the days (specifically, we'd have to rent the conference center for additional days to allow time to set up and tear down the Hall), but we'll see if we can solve those. (And, of course, there were plenty of people who liked the hours the way they were.)
This suggestion, about having explicit tracks in the program (we did have that) and have explicit time in each track to visit the Exhibit Hall (we did not do that) was very interesting. We hesitated to do this in the past because we wanted to avoid the effect of simultaneous specialized conferences that do not mingle, but perhaps EclipseCon is big enough now that we should consider it.
The Exhibit Hall had an Open Source Pavilion (sponsored by Business Objects; ten non-Eclipse open source projects) which 2/3rds of the respondents said we should do again. The other 1/3 said they didn't visit the pavilion. Sounds like a success!
75% of the respondents liked the new-for-2006 short talk format. In spite of my post-conference explanation that the short talks are meant to be surveys, 76% wanted longer talks with 15 minutes being the most popular answer and a mean value of 12 1/2 minutes. I feel that I could have worded this question better noting the tradeoff between the length of the short talks and the number of short talks; specifically, if we have longer short talks, then we will have less time for people to share their experiences with Eclipse. If we had gone with the longest time from the survey (30 minutes), we would have had only half as many speakers at EclipseCon!
74% of the respondents preferred the horizontal short talk format where all the short talks happen at the same time each day. In spite of the fact that this format is substantially more expensive in A/V costs because of the multi-monitor setup (and thus causes higher registration fees), we will probably retain the horizontal format for EclipseCon 2007.
This person even suggested having only short talks; an interesting idea to be sure...
We have not yet chosen a short talk length for EclipseCon 2007, although I'm leaning towards continuing shorter talks in order to allow more of the community the opportunity to speak. We're also considering an Eclipse Roundup like this person suggested.
The EclipseCon 2006 program was chosen by a volunteer program committee led by Tim Wagner. The program committee used the Eclipsezilla system to provide an open and transparent process for selecting the program. The goal was to allow the community to provide input about which tutorials, talks, demos, short talks, and posters they would like to see. Easier said than done, however, and the PC did not receive a large amount of community input.
The overall program was well received although there were a few rough spots. A large number of people complained about the tutorials being either ill-taught, disorganized, or just extended technical talks. That's unfortunate. We provided tutorials as a value-added extra-cost item but if people were finding that they are not worth the cost, then we shouldn't have them.
A number of people (e.g.) wanted the PC to verify the quality of the presentation and/or coach the speakers in advance in order to ensure higher-quality presentations. This kind of coaching requires a huge time commitment - much greater than the volunteer PC members can provide. Thus this kind of coaching/filtering is not going to happen.
What we are considering for EclipseCon 2007 is to push the content decisions even farther towards the leaves: rather than having a centralized program committee that makes all the decisions (with some community input), we will divide the program into segments, one per major Eclipse constituency (each major project, a few ecosystem tracks, regional or specialist tracks, etc). Each of these subgroups (the PMCs, the marketing committee, etc) will be responsible for filling their allocated slots at the conference. The theory behind this division of labor is that these people - the PMC members, the Member company Board reps, etc - know who's who in their area much better than any program committee could hope to. Thus they will know who has interesting content, who is a good teacher, who is a good presenter, and so on. They will be able to choose an excellent program around their area of interest and expertise.
This will satisfy people like this and this and this who requested tracks-in-advance to guide the program submissions. It may also help with the idea that someone should issue guidelines to the speakers - in this case, it will be the inviting PMC/group who will explain what is expected.
A number of people suggested ideas like "figure out which topics will be popular and then put them in the biggest rooms and/or in the mornings" (e.g.) Quite honestly, we tried to do that this year but obviously our judgment of which ones are going to be popular is not that great. If anyone has a good suggestion for how to figure out which ones will be popular, we'd love to listen to suggestions. Note: community voting for popularity is unlikely to work because (i) of low voter turn out (consider this survey, which had a Nokia Smart Phone as a prize, still only had a 26% response rate) and (ii) much of the content is submitted as late as possible, thus leaving no time for voting.
Suggestions from attendees that we will try hard to incorporate for EclipseCon 2007:
In summary, EclipseCon 2006 seems to have been pretty successful. There were a few glitches that we need to resolve for 2007, but with the help of you all (the entire community), in finding and selecting interesting content and interesting speakers, we should be able to have an even better gathering in 2007.
1 Most survey have a participation rate around 5%. Our free Nokia phone prize (thanks to Nokia for the generous donation) was probably a big reason for the higher rate. We note, however, that the quality of the survey responses was very high so the phone drawing seems to have motivated people to get started, but did not cause them to be brief.