All of the tests available in the Online Talent Manager were developed in-house by our own Bert Goos and André Tjoa. While whipping up a test might seem quite simple, it is a lengthy process that requires experience, insight, and inspiration. Developing a good, scientifically valid test can take up to two years and I wanted to share with you just what goes in to that process.
Part I – Idea
The process starts with an idea for a new test or we hear from our customers that they would like an instrument created to measure something specific. Our recent big ideas have been, “How do people learn most effectively?”, “How do people cope with stress?”, and “What kinds of work are interesting for a candidate?”
Part II – Brainstorming
The idea is on the table and now our test development team starts talking about all of the things that go into the concept they are trying to measure. They look at models developed by others, personal experience, and analysis of previous candidates in other tests (we are closing in on 220,000 tests delivered to candidates) to put together a new theoretical model. This theoretical model might take weeks to hammer out, but when complete, it will be used to construct the test itself.
Part III – Thinking up Items
Once the developers have a theoretical model, it is time to start hanging adjectives, actions, situations, emotional intensities, or whatever else is relevant in that model. Bert and Andre work separately on this first round of item-creation and then come together to compare lists. What comes out of this process is a list of items that they *think* will accurately reflect what they are attempting to measure. This list is usually pretty good right from the start, Bert and Andre are experts in this field, but no matter how strongly they feel about the list, it must be validated scientifically.
Part IV – First Validation
The goal of the first validation is to ask a large group of candidates to fill in the new questionnaire at the same time they fill in one or more previously validated tests. There is a published set of guidelines for how this process should be conducted from COTAN (Dutch Committee on Tests and Testing Affairs) and we adhere to those guidelines. After a large enough group of candidates have filled in the new questionnaire, a statistical analysis is performed to find out which items in the list are measuring what we think they should measure. This usually results in a shorter, but relevant questionnaire, but it might also result in the creation of new items and another round of validation. At the end of this process, we have a list of items that accurately measure the traits or behaviors in the theoretical model.
Part V – Testing and Normalization
The test is now valid, but all of the tests in the Online Talent Manager rely on Norm Groups to explain what the results mean. A Norm Group is made up of candidates that fit certain categories. A norm group can contain hundreds or thousands of candidate results, so this phase is continuous. As more candidates take the test, more specific norm groups can be created and we usually refresh these norm groups on a yearly or bi-annual basis.
Part VI – Continuous Development
The information that can be derived from a test is directly related to the size of the pool of candidates who have completed that, and other, tests in the Online Talent Manager. As the pool grows, it allows us to do statistical research into the connections between tests (inter-correlation), create new norm groups, and look for trends based on education, work experience, and other relevant categorizations. This continuous renewal keeps the tests relevant and up to date for today’s work force.
This process has been followed for each of the 92 tests in 4 languages (Dutch, English, French, and German) currently in rotation in the Online Talent Manager system. The expertise of Bert Goos and Dr. Tjoa have allowed us to grow the broadest and most comprehensive psychometric system available in the market today.
