The Certification Development Cycle

Both at the design and implementation levels, we follow a rigorous process to determine which competencies belong to a particular role and how we measure them. We draw from the best practices in measurement and psychometrics to design and implement each certification. In addition, each live certification undergoes a continuous quality assurance process to guarantee that each certification remains at the high-quality standards we aspire them to be at.

Certification design and development

You might have wondered how we determine the competencies measured in each certification. Based on an initial job task analysis and interviewing experts in the field, we develop a detailed certification specification in consultation with our certification advisory panel. Each specification contains detailed information on the purpose and audience of the certification, which competencies are being measured, and what the specific exam structure of the certification will be. Did you know that all specifications are publicly available? You can find them here:

• Data Analyst Associate
• Data Analyst Professional
• Data Scientist Associate
• Data Scientist Professional
• Data Engineer Associate

The Item Writing Process

For each certification, we work together with item writers who are professionals in the field. By doing so, we ensure that each item reflects the actual skills expected on the job. We have a rigorous item review process (see Figure 1 below), where items are reviewed by other item writers (SME review), internally by our certification content team (internal review), and finally, just before they are ready to be launched into an exam. For both the timed and practical exams, there are detailed guidelines on how to write items specific to the competencies being measured, what language to use (or not use), etc., to ensure all items are fair and devoid of bias.

Figure 1. The two-stage item review process.

Scoring Exams

Have you ever wondered how the timed and practical exams work and how they are scored? In the technical manual, we explain how we, for the timed exams, rely on item response theory and computerized adaptive testing to obtain a quick and accurate estimate of your skill level in each of the domains measured. Indeed, each question you get on the exam is tailored to your current performance, allowing us to efficiently determine whether you are at the level required for the certification you are trying to obtain. The score you obtain is then compared to a target score you must exceed to pass the exam. This score is calculated by simulating hundreds of exams of a hypothetical candidate that is exactly at the level we expect our candidates to be. Based on the distribution of expected scores for a qualified candidate, we then derive a target score we use during the timed exams.

Quality Monitoring

We continuously track quality metrics for all timed exams, including the pass rates and the reliability of each pool of items used in the timed exams. By doing so, we consistently provide reliable assessments of the domains we certify for.

While all timed exams are scored automatically, the practical exams are scored either automatically (Data Engineer Associate) or by human graders. To ensure grading is done reliably, we apply a double grading process where approximately 30% of submissions are graded by at least two graders, either both external graders or one external grader and one internal grader. We then compare how both graders score the rubric, and in case of disagreement, a third internal grader is assigned to rule out any disagreements. This process creates a continuous feedback loop for graders to ensure they are consistently grading against the rubric and awarding certifications to candidates that meet the qualifications.

Curious what more you can find in our technical manual? Go on and have a look!