A Piece of Cake: Assessment Development, Validation, and Standardization
Content (and cakes) by Dr. Allison Kimble-Cusano, Research Specialist
“I can’t help but see parallels between birthday cakes and the process of assessment development, validation, and standardization.”
Birthdays hold a special place in my family. As a busy mother of four, birthdays provide a chance to celebrate each of my children individually. They get to decide what we eat for dinner, who we celebrate with, and, of course, what kind of birthday cake will be served. Their choices range from homemade themed cakes to elaborate bakery creations, or even ice cream and donuts. Each year, before the smoke from the candles even clears, my children are already planning their next birthday cake.
Clearly, my children take birthday cakes very seriously. Choosing a cake is a year-long process that evolves as they grow and develop. Their taste buds change with new learning or emerging interests and sometimes even the most well-planned celebrations don’t turn out the way we expected. As an early childhood researcher, practitioner, and statistician, I can’t help but see parallels between birthday cakes and the process of assessment development, validation, and standardization.
Feeling confused? Let’s bake a cake together, and you’ll quickly understand what I mean.
Choosing a Cake = Defining Assessment Constructs
Before we begin baking, we need to choose a theme and flavor. Do we need a red velvet dinosaur cake, a classic chocolate sheet cake, a tiered snickerdoodle alien, or perhaps a Nutella ice-cream cake? This information will help us select the best recipe for our birthday creation.
Like us, assessment developers need to think ahead about the measures they create. They need to know the type and purpose of the assessment, what skills or abilities to measure, and for whom the measure is being developed. Like a recipe, this careful planning lays the groundwork for how the assessment will be structured and developed.
The Ingredients = Item Development
After choosing a recipe, we need to make sure we have all the ingredients. Typically, this means going item by item and checking for quantity and freshness. We may even need to take a trip to the store to purchase flour, eggs, or baking powder. Without the correct ingredients, our cake may fall short of expectations.
Measure developers must also pay careful attention to the individual test items included in their assessments. They may consult with community members and other experts to develop and select items that measure the intended constructs (i.e. language, cognition, program quality) and that are culturally relevant across diverse individuals and settings. This process is an important component of content validity.
Take a Taste = Development Sample
No matter how carefully we follow the recipe, sometimes the flavor is a little off. To help prevent this, it’s important to sample the cake batter. A quick taste can help determine if you need to add a little more sugar or stir a little bit longer. These little changes can have a big impact on your finished cake.
In assessment, a development sample or pilot study provides developers with an opportunity to test their finished product. They administer test items to a representative sample of participants and use statistics to take a close look at item performance. Developers retain items that correlate to one another and the target construct or constructs. They may also remove or reword items that are too easy or difficult or that perform differently based on the demographics of the test-taker (i.e. gender, race/ethnicity, socio-economic status).i
Bake at 350ºF = Validation Sample
Once our cake is in the oven, all we can do is watch and wait. It’s important for us to keep a close eye on our creation to make sure it isn’t under-baked or burnt and to check for doneness before we turn off the oven.
In many respects, administering the refined assessment to the validation sample is also a waiting game. However, just as we carefully monitor our cake while it bakes, assessment developers must closely observe the validation sample. They need to ensure that the sample includes a representative group of participants from the target population and that the assessment is given to enough individuals across various demographic groups, allowing for comparisons of overall and item-level performance between subgroups.ii
Cake Decoration = Validation Study
Decorating the cake is one of the most important steps in baking. At this stage we need be sure that our cake looks AND tastes great. We need to make sure the flavors complement each other and create an overall appearance that fits the theme and appeals to the tastebuds of all the partygoers.
Like our cake, measures need to meet developers’ pre-determined requirements. They use data collected from the validation sample to evaluate the reliability (consistency) and validity (accuracy) of a measure.iii They compare the assessment to similar measures (convergent validity), examine how well each item reflects the construct being measured (internal structure validity) and evaluate the extent to which the measure relates to specific characteristics such as age or disability (concurrent validity). Depending on the type of assessment, they may also evaluate how well a measure predicts a specific future outcome (predictive validity), and/or determine the measure’s ability to identify individuals that may need further assessment (screening tool sensitivity/specificity). As part of this process the assessment is also analyzed for consistency over time (test-retest reliability), across evaluators (inter-rater reliability), and across items and/or subscales (internal consistency reliability).
During the validation study, assessments are also evaluated to determine how well they work for different groups of people. This involves examining item bias and psychometric group differences to see whether the assessment fairly measures the same construct across demographic characteristics (i.e. age, race/ethnicity, gender, socio-economic status). This ensures that, like our birthday cake, measures seek to meet the diverse needs and expectations of their population of interest.
Cutting the Cake = Standardization
Cake cutting can be one of the most stressful parts of a birthday celebration. Before cutting we think carefully about the number of portions and work hard to make sure each piece is the same size. We may need to be strategic about who gets an edge piece or specific decoration and do our best to make sure everyone is satisfied with their slice of cake.
Standardization is the process through which an assessment is finalized, normed, and made available for use. This may include establishing cut off scores or rules about when to start or stop an assessment (i.e. entry points, baseline, ceiling). Standardization creates clear guidelines for administration and, like slices of cake, allows for the meaningful comparison and interpretation of test scores.iv
Everyone’s a (Food) Critic = Ongoing Revision and Validation
Bakers and assessment developers use feedback to improve their practice. As bakers, we use feedback to improve future cakes and adapt to different tastes. For assessment developers, feedback, in the form of ongoing revision and validation helps an assessment stay culturally and conceptually relevant over time.
In Closing
The next time you bake a birthday cake, take a minute to think about both the process and the product. Consider how each step influences the one that follows and reflect on how the experience mirrors the process of assessment development, validation, and standardization. Be sure to admire your creation, even if it doesn’t look or taste exactly as you had intended, and to learn from the experience. Then sit down, blow out the candles, and celebrate!
References
i. Kyriazos, T. and Stalikas, A. (2018) Applied psychometrics: The steps of scale development and standardization process. Psychology,9, 2531-2560. doi:10.4236/psych.2018.911145.
ii. Avşar, A. S. (2021). Comparing the automatic item selection procedure and exploratory factor analysis in determining factor structure. Participatory Educational Research. 9.10.17275/per.22.47.9.2.
iii. Martin, C. & Drouin Duncan, A. (2023). The IMPACT® framework: A guide to the key components of high quality early childhood measures. The Institute for Child Success. https://www.instituteforchildsuccess.org/resources/resource/impact-framework-early-childhood-measures/
iv. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in public health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149