Several years ago, as I was coming up on ten years of experience in project management, I decided to obtain professional certification. My reasons included a desire for professional growth, to interact with peers, to receive and in turn transmit knowledge, building contacts, and of course the cynical potential of personal gain. A bit of research confirmed that the Project Management Institute, which I'd heard of before, remained the 800 pound gorilla of the field. They offer the "Project Management Professional" credential, or PMP, which requires a college degree, 4500 hours and 36 months of project management experience, 35 hours of classroom training, and passing an exam.

In a series of upcoming posts, I will describe my experience with the PMP in detail. I will try to answer questions such as, Is the PMP helpful for employers? Is the PMP helpful for project managers? How practical is the knowledge covered in the PMP exam? How hard is the PMP exam? How well does the PMP satisfy its goal to "advance the project management profession and to recognize the achievements of individuals"? How well does PMI "promote a unifying influence in the advancement of Project Management"?

To start things off here in part 1, I'll summarize the controversy I saw surrounding the new exam released in 2005 with the Third Edition of the Project Management Body of Knowledge guide, or PMBOK.

---

I started thinking about getting a PMP in late 2004, but didn't do much research until early 2005. I discovered that a new version of the exam would be rolled out in September 2005. But before you can take the exam, you have to have 35 classroom hours. One option was to take a class at one of the local universities. San Diego State University and University of California San Diego both offer PMP programs, which are one or two-year affairs with a total cost in the range of $5,000 to $10,000. However, you can take a single class and pay in the neighborhood of a thousand dollars. But, it turns out that the San Diego Chapter of PMI offers specific Exam Prep classes that provide the required number of hours. And, because they are taught by volunteers who are themselves PMPs (seeking the 60 Professional Development Units, or PDUs, that PMPs must accumulate every three years to maintain certification), the classes cost only about $400.

The summer classes coincided with my annual July Vancouver vacation, which ended the chance of taking the old version of the exam. So I signed up for the fall class, six full-day Saturdays in a row, and after completing the classes registered to take the new exam in early 2006. During the class, many students were quite nervous about the new exam, and I found out why: The previous exam required 141 correct out of 200 questions, or 70%. The new exam still had 200 questions, and still required 141 correct, but 25 of the questions would be present solely for "testing the test", and would not count towards correct answers. So, on October 1, 2005, not only did all the questions change from "second edition" to "third edition", but the passing score rose from 70% to 81%.

Rumors passed around class included that the summer classes had forty or fifty people (we had about 12 regulars), that all available test slots in the San Diego area had been booked for months before the cutoff date, and that some students had flown to Nebraska and other underpopulated mid-West states solely to be able to take the old exam before the cutoff. The instructors said that this was part of an effort by PMI to "raise the bar" for new PMPs and make sure the credential didn't get diluted. As the class continued into OCtober, I heard more rumors, both from within class and without, that the percentage of test-takers passing the test on the first try was plummeting from 70%+ to 40% or even lower. (The test costs $405 (for members of PMI, itself a $119/yr cost, plus $10 for application and $30 for the local chapter), and $275 for re-takes.)

Eventually, I received an PMI email—not from PMI, but forwarded from an early test-taker:

...Before offering the new examination, PMI assembled a group of volunteers to help establish the passing score. Using a method known as the "Modified Angoff Technique" (a proven exam development method), a group of global PMPs in the summer of 2005 assessed each test question and independently evaluated the questions to determine their difficulty level. Their responses were then sent to PMI's psychometric (exam development) experts and averaged. From that information, PMI?s psychometricians recommended that PMI adopt a passing point of 81 percent (141 correct questions).

... After performing a statistical analysis of [actual test results], PMI and its independent psychometricians were able to make conclusions about the performance of questions as well as candidate performance ... PMI revised the passing score for the exam to 61 percent (106 correct questions). PMI then applied the new passing score to all examinations taken since 30 September 2005 by candidates who sat for the new exam. PMI is in the process of updating the candidates' records.

While we remain extremely sensitive to candidate and trainer concerns about such a change in the passing score, these considerations must be weighed in context of the overall purpose of the exam: to provide a consistent global standard that all practitioners must meet to ensure the credential is awarded to qualified individuals. ...

PMI understands that the changes to the exam and its passing score raise
numerous questions. ...

—forwarded email from Drew Ihlenfeld, PMI

It's hard to see such a drastic rescoring of the exam as anything other than a major failure of the exam development process. Of course, the retroactive rescoring effectively corrects the problem, but that's after-the-fact quality control, and it means that their before-the-fact quality assurance failed. And it doesn't address the wear and tear on PMP applicants, both those that were temporary failures and all of us in the months before and after the transition that had to make decisions in an atmosphere of uncertainty. For example, the email was dated 30 November, but it was weeks later before anything about it was posted on the website.

In the event, my classmates and I were very relieved to learn of the change in scoring, and I finally took and passed the exam on April 1, 2006 with a score in the high 70s. Nonetheless, I was somewhat bemused to recently read PMI's version of the fiasco in the 2005 annual report:

As part of best practices for exam development, PMI proactively reviewed data collected on the revised Project Management Professional (PMP) certification examination.

Driven by a role delineation study identifying volunteer-credentialed practitioner recommendations on how the skill sets for a PMP had evolved, PMI conducted this evaluation and revision based on the Institute's desire to accurately provide a consistent global standard that ensures the credential is awarded to qualified individuals, and meets the needs of PMI's stakeholders. Specifically, the attributes to "lead" and "direct" were identified as key elements of a more mature PMP.

PMI utilized proven examination development methods in revising the PMP examination, and conducted all appropriate due diligence to assess the applicability and difficulty of each test question as determine the overall percentage passing point.

Quality control measures implemented prior to and following the exam launch data indicate the exam is functioning as designed, including the fact that no inaccuracies in scoring and no malfunctions in the test's administration or its translations have occurred.

—PMI 2005 Annual report, p 23-24