A comprehensive survey and instrument design tool for faculty and researchers. Covers the full lifecycle of survey-based research: from construct definition and item writing through pilot testing, validation, deployment, and analysis of survey data.
Overview
Survey research is one of the most widely used methods across social sciences, health sciences, education, and business. Despite its apparent simplicity, designing a valid and reliable survey instrument requires systematic attention to construct definition, item wording, response format, sampling, bias mitigation, and psychometric validation.
This skill treats survey design as a scientific process, not an art. Every design decision should be justified and documented.
When to Use This Skill
This skill should be used when:
- Designing a new survey or questionnaire from scratch
- Adapting an existing instrument for a new population or context
- Writing Likert-scale items or other structured response formats
- Developing interview protocols or focus group guides
- Planning sampling strategies for survey research
- Conducting pilot tests and cognitive interviews
- Validating instruments (reliability and validity analysis)
- Selecting online survey platforms (Qualtrics, REDCap, Google Forms)
- Improving response rates and reducing bias
- Conducting cultural adaptation and translation of instruments
- Teaching research methods courses that include survey design
Does NOT Trigger
| Scenario | Use Instead |
|---|---|
| Analyzing collected survey data with weights, strata, or clusters (design-based estimates) | alterlab-survey-analysis |
| Analyzing interview or focus-group transcripts (coding, themes) | alterlab-qualitative-methods |
| CFA/SEM, measurement invariance, or IRT on the finished scale; a go/no-go check on whether a scale can be trusted | alterlab-sem-psychometrics / alterlab-ssci-measurement-gate |
| Delphi panels, Q-methodology, or other specialized social-science designs | alterlab-social-science-methods |
| Ethics/IRB applications, informed consent for survey research | alterlab-research-ethics |
For combining the survey with a qualitative strand use alterlab-mixed-methods; for hypothesis-test selection and power analysis beyond the instrument itself use alterlab-statistical-analysis.
Core Capabilities
1. Survey Design Process
The 10-Step Survey Design Framework:
Step 1: Define research objectives and constructs
What do you want to measure? What are your research questions?
│
Step 2: Review existing instruments
Has someone already validated an instrument for your construct?
│
Step 3: Define the target population and sampling frame
Who will you survey? How will you reach them?
│
Step 4: Choose survey mode
Online, paper, phone, in-person, mixed-mode?
│
Step 5: Write items and response options
Craft questions that are clear, unambiguous, and aligned to constructs
│
Step 6: Design survey structure and flow
Organize sections, add skip logic, manage survey length
│
Step 7: Expert review
Subject matter experts and methodologists evaluate the instrument
│
Step 8: Cognitive interviews and pilot testing
Test with a small sample from the target population
│
Step 9: Psychometric validation
Reliability analysis, factor analysis, validity assessment
│
Step 10: Deploy, monitor, and analyze
Launch survey, track response rates, clean and analyze data
2. Construct Definition and Operationalization
Before writing a single item, define what you are measuring. Map each construct to its dimensions, indicators, distinct-but-related constructs, and a nomological network so every item traces back to something specific.
Full Construct Mapping Template: see references/item_writing_and_scales.md.
3. Item Writing
Question Types and When to Use Them
| Type | Format | Best For |
|---|---|---|
| Closed-ended (single choice) | Radio buttons | Mutually exclusive categories |
| Closed-ended (multiple choice) | Checkboxes | Non-mutually exclusive categories |
| Likert scale | Rating scale | Attitudes, perceptions, frequency |
| Semantic differential | Bipolar scale | Evaluative judgments |
| Ranking | Drag-and-drop or numbered | Forced prioritization |
| Matrix/Grid | Likert items in table | Multiple items with same response scale |
| Open-ended | Text box | Exploratory, rich responses |
| Numeric | Number input | Precise quantities |
| Visual analog scale (VAS) | Slider | Continuous measurement |
Likert Scale — Number of Points
| Points | Trade-off | Use When |
|---|---|---|
| 4-point | Forces a choice (no midpoint) | Avoid social desirability midpoint clustering |
| 5-point | Most common, well-understood; central-tendency bias | Standard attitudinal measurement |
| 6-point | Forced choice with more granularity | Force direction with more options |
| 7-point | Greater discrimination; better for factor analysis | Established psychometric instruments |
Item-writing essentials — DO: simple/clear language; one concept per item (no double-barreled); specific time frames; match scale to stem; pilot with the target population; over-generate items. DO NOT: leading/loaded language; double negatives; assume knowledge; use absolutes; write overly long items; ask about hypotheticals when you mean actual behavior.
Full question-type examples, Likert labeling schemes, the complete DO / DO NOT rules, and worked before/after item revisions: see references/item_writing_and_scales.md.
4. Survey Structure and Flow
Organize the instrument as: welcome + consent → screening → main content grouped by construct (easy questions first, sensitive items mid-survey) → demographics at the end → thank-you/debrief. Use skip logic to hide irrelevant questions and route ineligible respondents.
Full Survey Structure Template and Skip Logic Design examples: see references/item_writing_and_scales.md.
5. Sampling Strategies
Probability Sampling (every member has a known, non-zero chance of selection; generalizable):
| Method | How It Works | Trade-off |
|---|---|---|
| Simple random | Select randomly from complete list | Unbiased, but needs a complete sampling frame |
| Systematic | Select every kth element | Easy, but periodicity risk if list has a pattern |
| Stratified | Random sample within population strata | Ensures subgroup representation; needs population knowledge |
| Cluster | Randomly select clusters, then sample within | Practical without individual list; higher sampling error |
| Multi-stage | Combine methods (cluster then stratified) | Flexible for large populations; complex to analyze |
Non-Probability Sampling (no representativeness guarantee):
| Method | How It Works | Trade-off |
|---|---|---|
| Convenience | Recruit whoever is available | Fast/cheap, but strong bias |
| Purposive | Select on specific criteria | Targets relevant subgroups; researcher bias |
| Snowball | Participants recruit others | Reaches hidden populations; biased toward the connected |
| Quota | Convenience sample within subgroup quotas | Ensures diversity; not truly random within quotas |
Size the sample with the proportion formula n = (Z² × p × (1-p)) / E² for descriptive surveys (adjusting for finite population and expected response rate), or a power analysis for comparative surveys. Worked sample-size formulas and the Python two-group power-analysis helper: see references/sampling_and_power.md.
6. Response Bias Mitigation
| Bias Type | Definition | Mitigation Strategies |
|---|---|---|
| Social desirability | Respondents answer in ways they believe are socially acceptable | Anonymous data collection; indirect questioning; validated social desirability scales (e.g., Marlowe-Crowne) |
| Acquiescence | Tendency to agree with statements regardless of content | Mix positively and negatively worded items; use forced-choice formats |
| Central tendency | Tendency to select middle response options | Use even-point scales (no midpoint); provide behavioral anchors |
| Extreme responding | Tendency to select extreme endpoints | Use more response options (7-point); provide clear anchor descriptions |
| Order effects | Earlier questions influence responses to later questions | Randomize item order within sections; counterbalance across respondents |
| Nonresponse bias | Systematic differences between responders and non-responders | Follow-up reminders; analyze early vs. late responders; compare demographics to population |
| Recall bias | Inaccurate recall of past events | Use shorter recall periods; provide memory aids; use event-specific prompts |
| Common method bias | Inflated correlations due to same measurement method | Use different measurement methods; temporal separation; marker variables |
7. Pilot Testing
Run a three-phase pilot before full deployment: (1) expert review for content/face validity (CVI thresholds: Item-CVI ≥ 0.78, Scale-CVI/Ave ≥ 0.90); (2) cognitive interviews (n = 5-10) using think-aloud and probing questions; (3) a quantitative pilot (n = 30-50) assessing completion, missing data, distributions, internal consistency, and item-total correlations.
Full phase-by-phase protocol with probe scripts and the quantitative-pilot checklist: see references/pilot_and_validation.md.
8. Instrument Validation
Assess reliability (Cronbach's alpha per subscale, corrected item-total correlations — flag items < 0.30) and validity across the evidence types below. Use exploratory factor analysis (Bartlett's test, KMO, eigenvalues, rotated loadings) to check internal structure. Report McDonald's omega alongside alpha: alpha assumes equal loadings (tau-equivalence) and understates reliability when they differ, which is why methodologists recommend omega (McNeish, 2018); alterlab-ssci-measurement-gate covers that check and alterlab-sem-psychometrics computes omega from a fitted factor model.
| Type | Question | Method |
|---|---|---|
| Content validity | Do items cover the construct adequately? | Expert review, CVI calculation |
| Face validity | Do items appear to measure the construct? | Target population review |
| Construct validity | Does it measure the theoretical construct? | Factor analysis (EFA/CFA) |
| Convergent validity | Does it correlate with similar measures? | Correlation with established instruments (r > 0.50) |
| Discriminant validity | Is it distinct from different constructs? | Low correlation with unrelated measures (r < 0.30) |
| Criterion (concurrent) | Does it correlate with a current criterion? | Correlation with gold standard, measured simultaneously |
| Criterion (predictive) | Does it predict a future outcome? | Correlation with criterion measured later |
| Known-groups | Can it distinguish groups known to differ? | Compare scores between groups that should differ |
Runnable Python for Cronbach's alpha, item-total correlations, EFA, and the three-phase pilot protocol: see references/pilot_and_validation.md. For CFA fit indices, measurement invariance, and the broader psychometric framework, see references/survey-methodology.md.
9. Online Survey Platform Comparison
| Feature | Qualtrics | REDCap | Google Forms | LimeSurvey |
|---|---|---|---|---|
| Cost | Institutional license (expensive) | Free for institutions | Free | Free (open source) |
| Skip logic | Advanced | Advanced | Basic | Advanced |
| Randomization | Yes (items, blocks) | Limited | No | Yes |
| Piping | Yes | Yes | No | Yes |
| Offline data collection | Yes (app) | Yes (app) | No | Yes |
| HIPAA compliant | Yes (BAA available) | Yes (designed for it) | Only under a signed Google Workspace BAA (paid plans); never on free accounts | Self-hosted: yes |
| API access | Yes | Yes | Limited | Yes |
| Data export | CSV, SPSS, Excel | CSV, Excel, SPSS, SAS, R, Stata | CSV, Excel | CSV, Excel, SPSS, R |
| Multi-language | Yes | Yes | Manual | Yes |
| Panel integration | Yes (Prolific, MTurk) | No | No | Limited |
| Best for | Complex academic surveys | Clinical and health research | Simple surveys, course evaluations | Budget-conscious complex surveys |
10. Interview Protocol Development
Build semi-structured interview guides with a scripted opening/consent, a warm-up question, main-question blocks organized by construct (each with probes), a closing catch-all, and a post-interview field-notes routine.
Full Semi-Structured Interview Guide Template: see references/qualitative_protocols.md.
11. Focus Group Facilitation
Plan groups of 6-10 (4-6 for complex topics), 3-5 groups per segment until saturation, homogeneous within and heterogeneous across. Assign moderator and note-taker roles, prepare a neutral environment, and use funnel-approach facilitation to manage dominant and quiet voices.
Full Focus Group Design Checklist: see references/qualitative_protocols.md.
12. Cultural Adaptation of Instruments
Adapt instruments across languages/cultures using Brislin's (1970) back-translation cycle (forward translation → independent back-translation → reconciliation → cultural review → cognitive interviews → validation) and the 10-step ISPOR cross-cultural adaptation guidelines.
Full back-translation flow diagram and ISPOR step list: see references/qualitative_protocols.md.
Best Practices
-
Start with constructs, not questions. Define exactly what you are measuring before writing a single item. Each item should trace back to a specific construct or dimension.
-
Use existing validated instruments when possible. Do not reinvent the wheel. Search the literature for instruments with established psychometric properties.
-
Pilot everything. Every survey should go through cognitive interviews and a quantitative pilot before full deployment. There is no substitute for testing with your target population.
-
Keep it short. Every additional item increases dropout risk. Include only items you will actually analyze. A good survey is as short as possible and as long as necessary.
-
Design for your weakest respondent. Write at an appropriate reading level. Test on mobile devices. Consider accessibility (screen readers, color contrast). Provide translations if needed.
-
Randomize item order within sections. This reduces order effects and helps detect careless responding.
-
Include attention checks. Embed 1-2 instructed response items (e.g., "Please select 'Agree' for this item") to identify careless respondents.
-
Plan your analysis before collecting data. Every question should have a purpose in your analysis plan. If you cannot say how you will analyze an item, remove it.
-
Document everything. Keep a survey design log recording every decision: why items were added, removed, or revised; pilot test results; expert feedback.
-
Protect respondent data. Use anonymous links when possible; store data securely; minimize collection of identifiers; comply with IRB requirements.
Common Pitfalls
| Pitfall | Why It Happens | How to Avoid |
|---|---|---|
| Double-barreled questions | Trying to be efficient; asking two things at once | Split into separate items; one concept per item |
| Leading questions | Researcher's hypothesis influences wording | Have a colleague blind to your hypothesis review items |
| Response options that do not match the stem | Copy-pasting from another survey | Ensure stem and response scale are grammatically and logically matched |
| Too many open-ended questions | Wanting rich data | Limit to 2-3 open-ended items; save depth for interviews |
| No pilot testing | Time pressure; overconfidence in item clarity | Always pilot — even a quick cognitive interview with 3-5 people helps |
| Ignoring mobile respondents | Designing on desktop | Test on multiple devices; avoid matrix questions on mobile (they break) |
| Low response rate | No follow-up plan; survey too long; no incentive | Pre-notify; send reminders (3-4 contacts); shorten survey; offer incentive |
| Neglecting psychometric validation | Assuming items are valid because they "look right" | Run reliability and factor analysis; report results in your paper |
| Convenience sampling reported as representative | Not understanding sampling limitations | Be honest about sampling method in limitations section |
| Cultural insensitivity | Assuming instruments transfer across cultures | Use formal adaptation procedures (back-translation, cognitive interviews) |
References
- DeVellis, R. F., & Thorpe, C. T. (2022). Scale development: Theory and applications (5th ed.). Sage.
- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.
- Fowler, F. J. (2014). Survey research methods (5th ed.). Sage.
- Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
- Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 263-313). Emerald.
- McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412-433.
- Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research. Journal of Applied Psychology, 88(5), 879-903.
- Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.
See also: references/survey-methodology.md for expanded methodology details.
Part of the AlterLab Academic Skills suite.