Validity

There are three major methods of estimating the validity of a data collection instrument.  The greater the degree of validity of the data collection device, the more confident you will be that the results you achieve reflect true differences in the scores of your subjects and not some random or constant error, the degree of  validity will reflect the degree to which we are controlling accounting for constant error.

The degree to which valid measurements can be achieved is directly related to the level of the study design.  Exploratory descriptive designs, by nature, have a low level of validation and must rely heavily on estimates of reliability.  Level II descriptive survey designs can achieve a greater degree of validity but must still rely heavily on reliability estimates.  Level III demands the highest degree of validity testing and uses reliability testing only to account for gaps in the attainment of validity. 

Just as control over the independent variable must increase with the level of design, so must control for error in data collection.   Methods of establishing validity of the measurement technique fall into one of three categories: self-evident measures, pragmatic measures, and construct validity.

Self-Evident Measures

These methods of establishing validity deal with basic levels of knowledge about the variable and look as an instrument’s apparent value as a measurement technique rather than at its actual value.  In other words, self-evident measures refer to the fact that the instrument appears to measure what it is supposed to measure.

Face Validity.

At the most basic level, when little or nothing is known about the variable being measured, the level of validity obtainable is called face validity.  “On the face of it...” merely establishes that the tool seems an appropriate way to find out what you want to know.  Looking at the questions you have developed to ask your subjects, you can say, “I think I will find out what I want to know by asking these questions.  It looks all right to me.” 

This is the extent of face validity.  It is the lowest level of validation and is used only when you are beginning to study a particular variable and have nor prior research literature to refer to.  If there is literature on the variable, either theory or research, then face validity is not sufficient.  If you have chosen to study a variable that has not been studied before, you will usually start with face validity, since it is the beginning step of the validation process.

Content Validity

Content validity is also a self-evident measure but involves comparing the content of the measurement technique to the known literature on the topic and validating the fact that the tool does represent the literature accurately.  You want to obtain an adequate sampling of the content area being studied. 

Content validity is frequently estimated from the review of the literature on the topic or through consultation with experts in the field who have become experts by having done unpublished research in the area.  After you have critically reviewed the literature, you construct your questions or instruments to cover the known content represented in the literature.

Content validity is a self-evident measure because it relies on the assurance that you can demonstrate an adequate coverage of the known field.  An expert should be able to judge whether or not the tool adequately samples the known content.  Researchers, therefore, frequently call upon experts in the field to verify content validity for newly developed tools.

In exploratory descriptive studies using participant observation, you may be in situations where you do not know either the setting or the population.  You assume that the persons you select to represent the population are knowledgeable about the content you are trying to elicit.  In this case, you assume that the members of a group or population have face validity as experts in their culture or social roles, and you try to further validate each person’s report by talking with as many experts as possible.  The more people you questions, the more content you will gain and the more depth of data you will have at your disposal.  “On the face of it” your informants appear to have face validity; you establish content validity of the data by cross-checking the answers with several informants until you are satisfied that the content is accurate.

Pragmatic Measures

Pragmatic measures of validity essentially test the practical value of a particular research instrument or tool and focus on the questions, “Does it work?”  “Does it do what it is supposed to do?”  Pragmatic validation procedures attempt to answer these questions.  The two types of pragmatic measures are called concurrent validity and predictive validity.

Concurrent Validity

Instruments that attempt to test a research subject on some current characteristic have concurrent validity if the results are compared and have a high correlation with an established (tested) measurement.  Suppose you had developed a checklist to measure pilots’ job satisfaction.  To validate this test, you would need to compare it with the results of an established job satisfaction instrument shown to be valid for pilots.  A high correlation between the results of the two tests would indicate concurrent validity for your checklist.

Predictive Validity

Instruments that accurately predict some future occurrence have predictive validity.  Measures designed to predict success in transportation programs fall into this category, as do aptitude tests.  They are designed to measure some current characteristic that is expected to predict something that will occur sometime in the future.  Predictive validity is established by measuring the trait now and waiting to see if the event occurs as predicted.  Once predictive validity has been established, the instrument can be used with confidence to discriminate between people on the basis of expected outcome.

Construct Validity

Construct validity provides the highest level of validation the most complex.  It deals with the validation of the construct (theory, proposition, hypothesis or principle) that underlies the research.  Here you are testing the theory that underlies the hypothesis or research question.  The term is derived from the fact that the characteristic under study is not a directly observable phenomenon but, rather, an abstraction or construct developed from observed behavior. 

To test the construct validity of a measuring instrument, you need to compare it with a number of other instruments that test for a similar construct.  Instruments that test for part of the overall construct should correlate highly with your new instrument.  Those that measure different, but related, theories should differentiate between yours and the others.