Online Talent Manager

All of the tests available in the Online Talent Manager were developed in-house by our own Bert Goos and André Tjoa. While whipping up a test might seem quite simple, it is a lengthy process that requires experience, insight, and inspiration. Developing a good, scientifically valid test can take up to two years and I wanted to share with you just what goes in to that process.

Part I – Idea

The process starts with an idea for a new test or we hear from our customers that they would like an instrument created to measure something specific.  Our recent big ideas have been, “How do people learn most effectively?”, “How do people cope with stress?”, and “What kinds of work are interesting for a candidate?”

Part II – Brainstorming

The idea is on the table and now our test development team starts talking about all of the things that go into the concept they are trying to measure.  They look at models developed by others, personal experience, and analysis of previous candidates in other tests (we are closing in on 220,000 tests delivered to candidates) to put together a new theoretical model.  This theoretical model might take weeks to hammer out, but when complete, it will be used to construct the test itself.

Part III – Thinking up Items

Once the developers have a theoretical model, it is time to start hanging adjectives, actions, situations, emotional intensities, or whatever else is relevant in that model.  Bert and Andre work separately on this first round of item-creation and then come together to compare lists.  What comes out of this process is a list of items that they *think* will accurately reflect what they are attempting to measure.  This list is usually pretty good right from the start, Bert and Andre are experts in this field, but no matter how strongly they feel about the list, it must be validated scientifically.

Part IV – First Validation

The goal of the first validation is to ask a large group of candidates to fill in the new questionnaire at the same time they fill in one or more previously validated tests.  There is a published set of guidelines for how this process should be conducted from COTAN (Dutch Committee on Tests and Testing Affairs) and we adhere to those guidelines.  After a large enough group of candidates have filled in the new questionnaire, a statistical analysis is performed to find out which items in the list are measuring what we think they should measure.  This usually results in a shorter, but relevant questionnaire, but it might also result in the creation of new items and another round of validation.  At the end of this process, we have a list of items that accurately measure the traits or behaviors in the theoretical model.

Part V – Testing and Normalization

The test is now valid, but all of the tests in the Online Talent Manager rely on Norm Groups to explain what the results mean.  A Norm Group is made up of candidates that fit certain categories.  A norm group can contain hundreds or thousands of candidate results, so this phase is continuous.  As more candidates take the test, more specific norm groups can be created and we usually refresh these norm groups on a yearly or bi-annual basis.

Part VI – Continuous Development

The information that can be derived from a test is directly related to the size of the pool of candidates who have completed that, and other, tests in the Online Talent Manager.  As the pool grows, it allows us to do statistical research into the connections between tests (inter-correlation), create new norm groups, and look for trends based on education, work experience, and other relevant categorizations.  This continuous renewal keeps the tests relevant and up to date for today’s work force.

This process has been followed for each of the 92 tests in 4 languages (Dutch, English, French, and German) currently in rotation in the Online Talent Manager system.  The expertise of Bert Goos and Dr. Tjoa have allowed us to grow the broadest and most comprehensive psychometric system available in the market today.

{ 0 comments }

I have been following a discussion on LinkedIn about the Big5 model and tests created to measure it.  The Big5 model is a description of personality that identifies 5 major personality factors.  It has a long history and the curious should read the Wikipedia article about this model.  Our own OP5 Test is measuring personality traits in the Big5 model and divides the 5 dimensions into 30 separate sub-scales, it is an intensive and very detailed test and used mostly by trained psychologists or specialized HR professionals.

The LinkedIn discussion went all over the place and it is in Dutch, but I thought I would share my contribution to that discussion here.  The question under consideration was this (paraphrased):

‘Why would I use a likert scale (each question asks the respondent to rate a phrase from “very applicable” to “not applicable”) test to measure personality, wouldn’t a candidate just mark ‘high’ on every question to get the most socially desirable results?’

So, I waded into the thread with my own take on how these results should be interpreted:

A well constructed likert style test (with a rating for each item from low to high) test will have ‘mirror’ items. That is to say, that on a significant portion of the items, a high rating will be negative for the scale it is measuring. For example; “I like to be alone”, if rated high, would have a negative influence on the ‘Extroversion’ scale. This means that a candidate blindly marking the highest level on all items will not get the result they expect.

This is also a part of why norm groups are used, especially norm groups based on the situation of the test. If a candidate is answering with the goal of getting a job (selection), they tend to be more positive about themselves than in a strictly development (coaching) situation.

If you are looking for the ‘shape’ of a personality, you should use a forced-choice questionnaire or use an ipsative analysis of the results. But that ipsative analysis gives no indication of the ‘degree’ of a personality trait. If you are looking for the ‘degree’ of a personality, you should use a likert scale test (as most Big5 tests are). The reason most Big5 questionnaires are likert scale is because some people really do have more or less personality (as a whole) than others.

A huge part of helping with the reliability of a test result is in how you communicate with the candidate. Putting the candidate at ease, explaining exactly how the results will be used and why honesty is important, and verifying the results with an interview or cross checking them against other tests are all good ways to ensure accurate results. (cross checking against other instruments is helpful because it is difficult to lie consistently)

That is also one of the reasons our professional users employ several tests from our catalog when looking at a candidate as part of a selection assessment. The different tests look at a candidate from different perspectives and we know, from over 90 thousand candidates, how those different tests normally relate to each other. If your personality is conflicting with your values and work style and communication style, then that raises a big red flag that will be addressed in interviews and/or phone consultations.

—

If you have a topic or question you would like to see addressed here and on our mailing list, please drop me a line r.still AT onlinetalentmanager.com.

{ 0 comments }

Andre Tjoa and Bert Goos wrote the first draft of a new test in March to look at how people learn.  4 months, several hundred candidates, a rewrite, and a lot of statistical analysis later, the Learning Styles Test was put into our customer test suite.  This instrument is now live for our Dutch customers  and we will translate and validate the test for English speaking candidates early in 2012.

At the HR Tech Conference and on HR discussion forums, there were a lot of companies marketing Learning tools and systems.  We now have some solid psychological research describing how people absorb information, a model for talking about it, and a test to measure it.

The Learning Styles Test is looking at how a candidate learns in terms loosely associated with the Octogram Model.  We were originally hoping to draw a direct line between how a person works and how they learn.  This didn’t work.  How a person works and how they learn are loosely correlated, they are not the same thing and a particular work style does not automatically predict how a person learns.  This means that the Learning Styles Test will remain a separate instrument in the OTM catalog.

So, how do people learn?

If you look at the graphic, you will see the names of the different styles starting at ‘Avonturier – Adventurer’ and moving clockwise around the circle.  The general model is still following the Competing Values Framework, so read up on that if you need a refresher.  This test is still only available in Dutch, so I will translate as we go!

Description of Learning Styles

Avonturier => Adventurer
Learns by doing and playing. Experiences learning as an exciting adventure and learns mainly by trying things for themselves. Driven by curiosity and wants to ‘jump in the deep end’ right from the beginning.

Koopman => Salesman
Social learner, likes to copy someone else. Wants learning to be a game. And if it is a competitive game, all the better!

Pragmaticus => Pragmatist
Learning must be an activity that delivers direct benefits. Prefers to get taught by experts with plenty of experience and wants to see what they are learning applied and in action.

Systeembouwer => System Builder
Whatever this person learns is placed in comprehensive theoretical framework. Information should be given to this person at a higher level of abstraction so that they can see how it ‘fits’ with everything else.

Perfectionist
Wants to learn things in a disciplined and systematic way. Learning requires lots of practice and repetition.

Kenniswerker => Knowledge worker
Attaches a great deal of importance to learning. Spends a lot of time analyzing the subject and also tries to learn more about the topic on their own.

Teamwerker => Team worker
Prefers to learn in a group so that others can provide feedback and share experiences.

Hulpvrager => Help me (literally a ‘help asker’)
Prefers to receive private lessons. Strong need for personal attention and guidance in the acquiring of new knowledge or skills.

So that’s a framework for describing how people learn.  And what is common in our research is that a candidate will have 1 or 2 strong learning styles and several middle of the road styles and 1 or 2 very weak styles.  A person with a weak score on, for example, Team Worker, would find a group learning situation boring and uncomfortable and would resist participating.  A person with a strong Team Worker score would be bored and stressed in an isolated learning situation.  Stress is bad for learning.

Even without the test, this information will hopefully give you a framework for talking about learning in your workplace.  Knowing how people learn will help you get the most out of your training budgets by connecting people with their optimal learning situations.

{ 0 comments }

There is a discussion happening on HR Toolbox under the heading: “Performance Management Metrics”.

One of the posters asked what is the most effective way to recognize and reward talent in their organization.  Here is a snippet from one of my posts in that discussion:

I would like to point you to two concepts/models that will help you keep your top performers and (hopefully) more fully engage your middle performers to reach a higher potential.

The first model is the Competing Values Framework. It is basically a way of looking at the various things people do in the workplace and trying to understand how individual employees like to work. We have learned through 10 years and 85k candidates, that the degree of match between their work style and their work requirements are a good predictor of success.

The second model you should become familiar with is the Schein Career Anchor model. Schein looked at *why* people work and developed (and validated) a list of what people value, what they want from their career. Schein’s list changed several times and we further refined it to a list of 10 career values that seem to have the most impact on success. Surprisingly, ‘Money’ isn’t one of them:

• Autonomy
• Creativity
• Entrepreneurship
• Competition
• Management
• Security
• Technical and Functional Specialisation
• Service and Commitment
• Collegiality
• Lifestyle Integration

The goal here is to find out what your employees value and make sure they get it. Do they need more opportunities to be creative? Are they inspired by competition? Do they need to be seen as the expert?

Both of these models will help you answer the big question: “How do I motivate my employees effectively?”

{ 0 comments }