IETF 121 post-meeting survey

The results of the IETF 121 Dublin post-meeting survey are now available on a interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 225 responses, 206 onsite and 19 remote. With 1592 registered participants, this gives the survey a maximum margin of error of +/- 6.06%. The number of remote respondents was much lower than previous surveys and the margin of error for that segment is +/-22.14%, so any results specific to remote participants are unreliable.

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Question changes since the last survey

For this survey we introduced a new matrix question about remote participation and added new lines to existing questions about the lunch break time of 11:30am, realtime transcripts and games night. As there was no social event, we again dropped the question about opportunities for social interaction.

As a one-off, we had a set of questions about the alldispatch experiment, which concluded at this meeting.

Actions taken following the last survey

For this meeting, we made the following changes, prompted by survey feedback:

• As this meeting was in a conference centre and we regularly get feedback that this reduces opportunities for socialising and ad-hoc interactions, we arranged for the break refreshments to be in one room instead of split across floors, and supplied a bar two nights of the week on premises.

Satisfaction

Overall satisfaction is 4.30, a good result. With some key exceptions, the satisfaction scores remain high.

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)

Remote participation

We once again asked people why they participated remotely and asked if those who participated remotely would have wanted to participate onsite.

Putting the two together continues to show a consistent proportion of remote participants would have participated onsite but were unable to do so because they did not have the funding.

However, remote participation itself gets high satisfaction scores for the various components (an overall satisfaction question was accidentally omitted).

Sources of funding

For the second time asked the question "What was the primary source of funding for your onsite participation?", with the overwhelming majority (72.65%) funded by their employer, and the next highest segment (10.76%) using their own personal funds.

New participants

We regularly include a set of questions for those who identify themselves as new participants and these have been used to drive large changes to the new participant program over the last two years. We will provide a more detailed analysis of the results of these questions in a future blog post.

Areas for improvement

Break food and beverage

The satisfaction score this time around (2.88) is Very Poor and by far the worst score we've had for any area of the experience under our direct control. I can only apologise and pledge that we will ensure that the various issues we experienced this time are not repeated.

Breakout seating

Once again we were constrained by venue health and safety rules from providing the level of seating people want near to the breakout rooms. This is one of the last venues that was negotiated without an extra seating clause so we expect this to become less of an issue for future meetings.

Welcome reception

We've now had a Farewell Reception for three meetings and that has had a consistently higher satisfaction score than the Welcome Reception, indicating that the Welcome Reception could be improved. We welcome any suggestions on what it is lacking and how it could be improved.

Opportunities for social interaction

We are in a global economic environment where meeting hosts have budgetary constraints that mean they cannot fund a social event and our efforts to address this, avoiding a significant increasing in expenditure, have been so far limited to the introduction of a game night. This is another area where we welcome suggestions - should we do something to help people meet up for dinner, recommend local events, or something else?

Side meetings

While the larger issue of side meetings remains problematic, one consistent request has been for better support for them on the meeting agenda. The reluctance to do this in the past has been that this would imply they are an official part of the agenda, but we will look to see if there is a way to do this that clearly addresses their status.

Areas we might not be able to improve

Conference centres

When we meet in conference centres we regularly receive very divided feedback (across in-person and the survey) with some strongly in favour and some strongly against. Our policy is to prefer a one-roof approach, but there are multiple countries/cities we will only be able to meet in if we meet in a conference centre and so we will continue to switch between the two.

Cost of accommodation

Over the last several years the accommodation market has changed significantly with the advent of short-term rental aggregators, and hotel booking aggregators. As seen in this survey, participants are choosing accommodation from a wide range of options and are distributed across the city. Before this change, we used to contract a lower rate secondary hotel but reach the stage where this booking was underused and risked us paying contractual penalties, and so we stopped.

We do, on occasion, book a meeting in a city that is less of a travel hub or less of a destination than others and so has lower cost accommodation across the board, but in those cases we often receive feedback that the extra travel hop was too much or that sponsors are less interested in supporting such a meeting.

Conflict avoidance

As noted previously it is likely that we are simply not able to do any better given the number of sessions and the limited days in which we can meet. The IESG has made the Friday afternoon experiment a permanent feature in order to help with this, but that in turn has brought multiple complaints and a lower satisfaction score than other aspects of the meeting.

Cost of registration

The IETF is a charity that relies on donations, sponsors and registration fees, and so has to set those fees high enough to keep operating. This is offset by this being the only part of the IETF where there is a participation cost, and by the high quality remote participation and unlimited remote fee waivers. To address concerns, we are talking to sponsors about a formal grant program, but in the current economic climate this is not progressing quickly.

Individual comments

The individual comments covered a number of key themes, most of which have already been covered above, leaving the following:

• Coffee queue. This venue was booked before we introduced barista coffee and when it came to arranging it, they would only allow it if we paid a high fee, and so we were unable to have the same coverage as at previous meetings.
• Cut short farewell reception. This is still a new event that is subject to change. We will consider how to address this feedback.
• Other comments. Other comments will be passed on to the appropriate people to consider.

And finally

Thank you everyone who responded to this survey, your feedback is much appreciated.