Our Client’s Request
Our government client is very hierarchical and authoritarian. As a result, the organization’s middle level management is accustomed to working within this highly structured environment with a clearly defined mission and task mandate. In their system, employees know exactly what they are supposed to do, and what the consequences will be if they don’t work as expected. While this style fits the organization’s mission and culture, there is one highly unique position within the organization that requires a completely different set of skills.
Our client has been hiring people for this unique role using a process that doesn’t quite provide a full picture of the candidates’ fit and doesn’t cover all of the job requirements. As a result; the incumbents keep on having issues finding success, resulting in performance incidents that have critical consequences for this important position.
For this role, our client is looking for candidates who can:
- Work in an isolated environment with little to no supervision
- Influence people who do not work for them, and gain support for the organization’s mission
- Succeed in a challenging location without a large support system
Unlike most of their positions, this particular leadership role is isolated from other leaders and is located away from the structure and support of the main organization, requiring a unique and highly specialized skillset that is different from other critical positions in the organization.
Given that this position is high profile and central to the organization’s strategy, it’s extremely important for our client to have the right people in this role. However; it’s difficult for the organization to identify the most qualified candidates because the job is so different from the other leadership roles they typically have. Adding to the challenge is that an internal candidate’s success in a highly structured, hierarchical system may not predict their success in this ambiguous, “wild west” leadership position, away from the typical checks and balances that accompany senior leaders in this organization.
What We Are Doing
DCG was tasked with helping the organization identify the right types of internal candidates to move into this a-typical, ambiguous, and international leadership role. This particular role has a large component of leadership with influence, especially across other cultures, and these types of skills are not typically required of other leaders in the organization.
Our combined team of organizational development consultants and industrial/organizational psychologists is taking a blended approach to help our client ensure they are getting the right people in the roles. We began our work by spending time getting to fully understand the client at large, and the role in particular.
Our organizational development consultants bring a deep understanding of what makes an organization work, as well as how to structure leadership positions like this one for success. Complementing the expertise of our Organizational Development (OD) team, our Industrial/Organizational (I/O) psychologists consider the personal characteristics and individual differences that will help employees be successful within the system, especially in an unorthodox role like the one our client has been struggling with.
Step 1: Defining Success
We first made sure that we understood what success in this role looks like. We asked our client many questions like:
- How do you know when someone is successful in this role?
- What about unsuccessful?
- What are some of the things that top performers do (or don’t do) in this role?
- What are some of the things that poor performers do (or don’t do) in this role?
Although these are simple questions to ask, the answers reveal information about what I/O psychologists call the “criterion space” or essentially whether or not someone is successful on the job. Once we really understood what job performance meant for these leaders, we began our research to determine which types of people would be able to develop into high performers.
Step 2: Identifying Attributes
Once we had a clearly defined criterion space, our next step was to figure out what types of “attributes” or personal characteristics people in the organization needed to possess to be successful in this role, and what might be some “red flag” qualities that we want to help our client avoid.
We found that there were several key characteristics required for success in this role, some of which were:
- Works with and influences partners to work toward the goals of the US organization at home and abroad
- Thrives under conditions of high ambiguity
- Makes ethical choices in difficult environments with limited oversight
To help us with both steps 1 and 2 above, we spoke to many subject matter experts, both currently in the role, and in positions that interact with the role. We used a mix of qualitative (semi-structured interviews) and quantitative (survey data) techniques to deepen our understanding of the role.
In addition to our work with internal subject matter experts, we also looked at similar roles in the available literature and at other, related government and private sector organizations. This resulted in a broad literature review that supplemented our work with the subject matter experts and helped us come up with some keys to success for the organization.
Step 3: Measuring Attributes
Once we compiled our list of attributes, some of which were:
- Ethical decision making
- Cultural Flexibility
- Negotiation
- Tolerance for Ambiguity
- Conscientiousness
We began to think about how we could assess and measure these attributes in the pool of candidates. In order to determine the best way to measure these attributes, we took a rigorous, multi-step approach to develop a reliable and valid assessment of those attributes that our client could use to identify the most qualified candidates in the applicant pool. We first developed a prototype by identifying and/or modifying pre-existing attribute measures (either that the client had on hand or that were available in the public domain) and developing new measures where none could be identified. We are currently in the process of completing measure development and will soon be moving into the second phase of the project, validating the ability of our measures to predict job performance.
Step 4: Validation Studies
Our team of I/O psychologists will be evaluating the effectiveness of our assessment by conducting a “concurrent validation.” In this phase, we will administer our assessment to individuals who are currently working in the role and collect data on their performance. We will then conduct statistical analyses to determine if our assessment is able to differentiate between high and low performers. We will also conduct analyses that will inform the reliability, or consistency, of the different attribute measures in the assessment. By linking the measures in the assessment to performance on the job, we will be able to determine how well our assessment predicts performance. Finally, the concurrent validation provides an opportunity to examine the fairness of the assessment battery as we will conduct statistical tests to make sure there are no substantial differences in scores based on gender, age, race, etc.
Taking all this information together, we will be able to make recommendations to our client for revising the assessment and setting up a longitudinal validation phase, also called predictive validation. The predictive validation will provide a more realistic picture of how effective the assessment tool is at predicting performance, as it tracks actual applicants for the role over a certain period of time. We will administer our revised assessment to applicants for the position and collect data on their performance at multiple milestones. We plan to conduct several statistical analyses to determine how well the assessment predicts hiring decisions, training performance, in-role performance and other important outcomes.
Step 5: Creating the Final Product
By taking into consideration all of the statistical analyses, we will be able to help the client determine what the final assessment tool should look like for operational use. We will also use the analyses we conducted as a basis for creating a scoring rubric. By assigning greater weight to the attributes that were most predictive of performance, the applicant’s overall score will be more meaningful. We plan to use I/O psychology best practices of conducting both a concurrent and validation study to determine how well the assessment predicts performance. At the conclusion of our project, our client will be able to use this validated assessment with confidence, knowing that it will help get the applicants into the role who are most likely to be successful.
The Outcome
Our client will have a rigorous, scientifically based selection instrument that gives them a more complete picture of a candidate’s fit for the role and their likelihood to perform well. They will be able to use a validated assessment tool, in conjunction with other selection criteria, to make an informed and fair selection decision. With its scientific foundation, our client’s leadership can trust the method we created and applied to help them get the right candidates in this unusual job, and job applicants will know that they are working through a rigorous but fair assessment process to be selected for the job.
Let us know how we can help your organization develop an individualized selection system.
Contact Us: inquiries@desmondcg.com
PhD, Industrial-Organizational Psychology