I heard this recently after the results of our captain's promotional process was completed. I actually heard variations of it more than once, and sometimes with a tremendous amount of conviction and passion. Maybe even more conviction and passion than was used to prepare for the test. That is my explanation for the poorer scores.
As head of the training division, I was tasked with preparing and executing a promotional exam. Preparing for and taking an exam for promotion is pretty involved. I knew that from my side, there was also a great deal of preparation - this was the third one I had done in it's entirety, and had participated in others on both sides of the table. I've got a pretty good handle on the process. What I always wonder (and I hope I'm not the only one): do we have a method to choose the best and brightest? I'll describe our current one below. Over my time I've seen, and heard of other methods of promotion, and ponder about the effectiveness of any to pick what is a very important leadership position.
There was (and maybe this still exists) the seniority method. Someone left and created an opening which was filled automatically by the next most senior member. Then (this was my first as a candidate), there was a basic tactical scenario (no computers or virtual reality at that time) followed by a short interview with the chief (fairly small department). Variations on that theme evolved to include interview panel boards, more involved tactical scenarios, written exams of varying quality and quantity, and seniority was sometimes also included.
Next, and what we use currently, was the full assessment center. This is how ours rolls: a comprehensive written exam; a computer generated full tactical practical exam; a role-play exercise, and finally a formal interview board. Most departments in my area use something similar. The down-and-dirty details differ somewhat, but overall that's the program.
My nagging worry is what I stated above - are we picking the best candidates, and is there a better way? Is this process fair? We work very hard to insure that every candidate takes exactly the same test, is provided with exactly the same scenarios, and the same assessors work for all of the candidates. Check. The process is stellar, or sucks, in the same way for each candidate.
We try to measure, as best we can, the attributes we seem to agree leaders should have: leadership, management perspective, job knowledge, and more. I think that overall, we do a pretty good job of allowing the candidates to demonstrate these attributes in a fantasy world built by knowledgeable and well-intentioned personnel. In a role-play activity, we put these folks in a scenario with a handful of the daily problems faced by a company officer, and let them actually solve them through communication channels (written, computer and human interactions) and actions. Tactical scenarios are built from real life incidents, and is pretty straightforward. Written exams are trivial pursuit exercises. Some folks retain and can regurgitate information with little effort. Others may have to put in a tremendous amount of effort (that usually will pay off). Either of those two personalities can stay competitive in the overall process. The oral board gives a candidate the opportunity to really demonstrate what's inside their head, and if they've really got the concepts of leadership and organizational perspective need to carry out the position.
And yet...we all know there are some who do very well on these comprehensive extravaganzas, yet turn out to be horrible leaders. And we all know those informal leaders, who can't perform in the fantasy world of testing. Yet, they would almost surely take any group positively forward in any and all scenarios - the individual you really want to work for. No process is immune from failure, especially when dealing with personalities and stuff that can be somewhat subjective. But when that dysfunctional individual ends up in charge of a fire crew, it shakes my confidence in the promotional process a little bit. And when that other stellar employee doesn't get their chance to excel, and maybe never will, it can be even more distressing. I typically counsel candidates, both before and after the process (and this is especially for the poor performers who insist they did better), that there is a certain amount of gamesmanship in these kinds of tests. In order to do well, not only does one need to know how to actually do the job (actually lead people, understand tactics and technical stuff, etc.), but communicating what you know and how you would use it for a small and focused audience is the only way to score well. That requires what I call the need for test-taking strategies to complement actual usable knowledge in order to obtain a promotion. I see our failure to weed out the worst candidate is that the best natural actor/orator/presenter can weave his or her way through the testing process to score high enough to be promoted into an area where there is the potential to do some organizational, and human, damage. Here's also where we have the potential to get the inexperienced but highly intelligent firefighter who tests themselves into a demanding position that requires some emotional maturity, and some actual street experience. My feeling is that seniority as any end-all be-all can't be effective. However, the smart but ill-prepared 25 year old, with seven years on the job, and only four working fires and little experience to lead in tough circumstances, will struggle mightily for awhile - maybe permanently. The demands of the job can make them irrational, ineffective, organizational anchors, and sometimes worse. Of course, some also grow into the job, and become great. What ends up humorously, is that that same actor/orator that scores poorly thinks that actual knowledge of doing the job well should be able to be replaced by a great performance art.
Unfortunately, we can't develop a parallel universe where a department holds a full-blown assessment center for promotion. Then in that other universe, picks a guy based on seniority (or some other very simplistic method), and then examines the long-term outcome. What would be the scientific results? Would the big deal tests have a higher percentage of successful leaders? Or would that guys that happened to stay around the longest provide the best group? I'd like to believe that we currently are using the best method, but I still can't help but wonder sometimes. I've seen good and bad leaders end up at the top and the bottom of all the tests listed. Pretty sure anyone reading this knows, or is, someone who falls somewhere in the above descriptions. I guess all we can do is continue to tweak the process, and give our best reasoned efforts. Organizational support and development of new leaders is critical, no matter what. That's the back end of the promotional game, and maybe a future blog post. Now, if anyone knows anything about creating, or at least observing, a parallel universe... please contact me ASAP.