Criterion-related validation studies typically include the following steps (Crocker & Algina, 1986):

1. Identification of an appropriate criterion and a method for measuring it. For a mechanical aptitude test, the criterion might be subsequent job performance data.
2. Identification of an appropriate sample of examinees representative of those for whom the test will eventually be used.
3. Administration of the test and recording of each examinee’s score.
4. Measurement of performance on the criterion for each examinee.
5. Determination of the strength of the relationship between test scores and criterion performance.

There are two types of criterion-related validation: predictive and concurrent.

Predictive Validation Study

Predictive validity refers to the extent to which test scores predict criterion measurements that will be made at some point in the future. In the case of validating a paper-and-pencil mechanical aptitude test, applicants take the test but their test scores are not used to make hiring decisions. Over time and after a sufficient number of job applicants have been tested, job performance data is gathered and correlated with mechanical aptitude test scores. A positive correlation would be evidence of the predictive validity of the mechanical aptitude test.

Concurrent Validation Study

Concurrent validity refers to the relationship between test scores and criterion measures made at the time the test was given. In this case, a sufficient number of current job incumbents take the paper-and-pencil aptitude test. At approximately the same time, job performance data is collected on the incumbents and correlated with their mechanical aptitude test scores. A positive correlation would be evidence of the concurrent validity of the mechanical aptitude test.

References:

Crocker,L., & Algina, J. (1986). Introduction to classical & modern test theory. Florida: Harcourt Brace Jovanovich College Publishers.