Tips for Creating Good Surveys

Office of Educational Assessment (OEA)

Tips for Creating Good Surveys

Anyone involved in creating or editing a course survey will find helpful pointers here for putting together a good survey.

Creation and Implementation of Educational Surveys

I. Planning Phase
Identify and record specific objectives for the survey before composing the questions. This will help focus questions on important topics that truly require answers. Surveys often ask too many questions that provide unnecessary data.

Questions should be written to test or confirm specific hypotheses.

If one likely answer to a hypothesis is strongly suspected, careful attention must be given to eventually write an unbiased question (see below).

Carefully plan enticements or requirements for the survey participants. Examples include withholding a final grade until a student completes the course evaluation, giving bonus points for timely completion of the survey, etc. The aim is to reach as close to 100% participation as quickly as possible, in order to improve the validity of the group results.

II. Survey Structure
The survey should begin with a brief overview of the objectives of the survey.

Length is important. While there are no absolutes, ideal length is perhaps 15-30 questions. It is difficult to assess the validity of the final questions in an overly long survey, as enthusiasm and interest wane. If a survey being designed is too long, assign each question a priority – is it "essential" to know this, would it be "nice" to know the answer, or does it fit the category of "who really cares?"

The wording of questions requires careful thought. In general, questions should be brief, direct, unambiguous, and cover a single topic. Questions should be written in neutral language. Biased or judgmental wording (such as should, ought to, bad, wonderful, etc.) would obviously lead responses. For example, contrast the following: "the faculty were available to help" vs. "the faculty should have been more available to help". The ideal question stem does not reveal the writer’s pre-survey hypotheses.

The "set" of questions also needs attention. Questions should be grouped into subsets, with headings to orient the respondent. All of the questions should fit together in a logical, orderly, thematically holistic manner. Questions from "left field" are distracting.

The order of questions matters; an early question about a contentious subject may slant the remaining answers about more neutral topics. Therefore, early questions should be less controversial and preferably peak interest.

Research suggests that demographic questions, perhaps surprisingly, are best left for the very end. They clearly require the least thought, thus allowing the respondents enough energy to tackle more complex questions. Furthermore, questions that are perceived as "personal" may make students more defensive and believe that their anonymity is being violated, thus altering their subsequent responses. While demographic information can be used for sub-group analysis (what do female students think vs. their male colleagues, etc.), it is advisable to only collect data that are going to be analyzed.

The best questions for group analysis use a Likert-type intensity based scale. A concise, easily understood question stem should be followed by a number of specific responses. Likert questions may have from 3-10 scales arranged along a continuum of responses. While there is no dogma about this, 5 responses is perhaps the best with a wide enough spread of responses. An odd number of scales also allows the middle scale to be a true neutral response. Seven or 9-scale questions permit greater discrimination at the expense of extra time needed to respond.

It is important to maintain consistency amongst scale questions, so that all worse-to-best scales run in one direction (left to right or right to left). Research suggests that a scale with the best response to the left provides a higher mean response than the same question arranged worse-to-best (called the primacy effect, or the tendency of people to favor the left side of the scale). So, best-to-worse will yield better results, while worse-to-best may give a lower mean with a wider standard deviation.

As a corollary, open-ended short answer questions provide richer, more personalized responses. They are unfortunately harder to analyze, since group summaries can only be made after time-consuming content analysis. Survey authors are therefore advised to carefully think about limiting the number of short-answer questions unless they are prepared to spend considerable time in assessing group consensus.

The survey should end with a brief description of how the data will be used. Participants are well aware that the data is most useful for groups that follow, but a sincere thank you and reassurance that someone will be actually reading and analyzing the data is helpful to encourage survey completion in the future.

III. Confidentiality Issues
It is essential that survey participants, especially students, feel that their anonymity is maintained. If not, surveys will either not be filled out, or will have self-censored data.
Equally essential is the need to track participants who complete or fail to complete the survey. This prevents duplication; allows tracking of those who habitually fail to complete surveys; and permits more sophisticated analysis of an individual’s responses over several surveys.

The best way to accomplish this is with a list of "unique identifiers". Each participant is assigned a unique code, which they must enter in order to complete the survey. Ideally, only a central or neutral group (i.e., the Office of Educational Assessment) knows the code and not to those who will use the data. Anonymity is maintained and tracking is possible to allow course leaders to follow trends in responses over sections of a course. The potential downside to this is if participants forget their code, or believe that entering their unique identifier will eventually allow someone to match answers with names. Trust is essential for this system to work. [As an aside, on rare occasions surveys do contain unprofessional or egregiously inappropriate comments. With due process being followed, through the Dean’s office for example, the code can be broken. The mere knowledge that this is a possibility functions very effectively in preventing the inappropriate comments from being written.]

IV. Implementation Issues
The greater the percentage of students completing a survey, the greater the validity of conclusions drawn from the data. The ideal is obviously to have a 100% completion rate. This is best accomplished by making completion of the survey a course requirement.
The survey can be on-line, or in paper format. There are advantages to each. On-line surveys allow flexibility of completion times. Survey software packages facilitate rapid summarization and analysis of data, with quicker turn-around time for course leaders. Paper surveys are more cumbersome to process, but are easier to present to classes. Course leaders can devote curricular time to the completion of the surveys, thereby assuring nearly 100% completion all at the same time.

Attention should be paid to the "carrots & sticks" used to encourage completion, especially for on-line surveys. Many courses withhold grades until the surveys are completed. Creative enticements for timely completion should be considered as an alternative.

V. Post Survey Issues
Plan to give feedback. Students report a drop-off in enthusiasm in completing course evaluations because they never receive feedback about the surveys. It is strongly encouraged that summary reports be created for each survey, and widely distributed to participants and relevant faculty members. Such a report should obviously have deleted out any information that identifies an individual student or faculty member.

The objective, summarized data from a survey should be analyzed and used by course directors and program coordinators as they redesign curricula. A well-designed survey with a high completion rate should clearly identify areas of strength and those needing attention. As an observation, such composite data are often underused; while anecdotal, usually verbal comments from a small subset of participants are given too much credibility.

Whenever possible, survey results should be compared to other data, such as surveys from previous years, other courses, other schools, published studies, etc. As an example, a survey of one section of a yearlong medical school course revealed that 25% of the class thought that the course objectives were poorly met. This appears to be a modest number until it is compared to data from all other sections showing <5% of students felt the same.

Summary results should be reviewed to look for possible bias. Common sources for bias in questionnaires include:

Sample bias. If there is not 100% participation, is the group completing the form representative of the entire group?

Leading or poorly written questions in the survey.

Participant confusion in answering a given question.

Untruthful/Misrepresentative answers. While this is immeasurable, active lying is probably rare. What is undoubtedly more common are rote answers from disinterested or fatigued respondents ("I will just fill in the middle category to save time"). This reinforces the need for a succinct and anonymous survey.

A. Ardolino
November, 2001

Office of Educational Assessment
School of Medicine
University of Connecticut Health Center
263 Farmington Avenue, Farmington, CT 06030-1925
Tel: (860) 679-4590, Fax: (860) 679-1832