Assessment Design Manager
JOB PURPOSE:
- Provides exam content development support for all ABO assessments.
- Oversees program evaluation and improvement for all ABO assessments.
- Develops, implements, and evaluates oral examiner training.
- Oversees content outline reviews and revisions for all ABO assessments.
- Contributes to organizational learning and development.
- Conducts research on assessment innovations.
POSITION RESPONSIBILITIES:
Assessment Design and Exam Development (45%)
- Develops the onboarding and training curriculum for ABO item writers and examiners
- Oversees the training of the oral examiners:
- Develops and communicates the training schedule
- Ensures all content within the training materials (manuals, slide presentations, video recordings) remains current and coordinates updates when needed
- Secures and onboards the training presenters
- Organizes and shares the virtual meeting logistics (e.g., Zoom links, calendar invitations)
- Conducts, records, and runs training webinars/meetings.
- Collaborates with the operations staff on the timeline and language for volunteer communication campaigns
- Oversees content outline and test specifications development and revisions
- Collaborates with the Exam Development Manager on item bank evaluation, identifying gaps, and developing workplans to support new item generation
- Collaborates with the Volunteer Manager on the evaluation and training of all item writers and examinations.
- Develops criteria for examination form assembly for all examinations with the Director of Psychometrics and Assessment
- Assists the Director of Psychometrics and Assessment with key validation for all ABO assessments
- Develops and supports policies and procedures related to examination security and item exposure
- Collaborates with the Certification Experience team in development of materials for candidates related to the exam format, rules, and expectations
Organizational and Program Management (20%)
- Oversees the evaluation of all assessment programs including but not limited to:
- Identifying goals for item retention and performance
- Designing, disseminating, and analyzing results of surveys
- Coordinates assessment design changes with all relevant teams
- Assists in evaluation of ABO assessments with compliance with professional standards, including but not limited to NCCA and ABMS Standards.
Research and Innovations (20%)
- Conducts literature reviews and research related to assessment design (including but not limited to the use of AI in item generation and scoring, alternate item types)
- Develops process improvement pilot projects
- Collaborates with other boards and experts on exam design research and evaluation
- Coordinates research meetings with internal and external researchers
- Co-authors peer-reviewed research
Administration (10%)
- Assists with/attends Initial Certification Committee meetings and/or other ABO Board and Committee meetings as requested
- Supports basic data analyses, writing of reports or minutes, and/or other support tasks related to exam development
- Conducts availability polls and schedules meetings for various teams (internal and external)
- Attends staff meetings and collaborates with other teams
Professional Development and Training (5%)
- Attends and presents at professional conferences (e.g., ATP, AERA/NCME, ABMS)
- Provides education to staff, Board of Directors, and others
REQUIRED skills, experience, and professional characteristics
- Advanced degree (MS, MEd) in Educational Psychology, Applied Psychology, Measurement, or related field
- Able to learn new software applications and assimilate new material quickly.
- Exhibits willingness to learn new skills and participate in professional development.
- Communicates accurately and effectively, internally and externally.
- Uses various applications to coordinate program tasks such as file management, scheduling, evaluation tracking, and survey tools.
- Proficient with Microsoft applications (Word, Excel, Adobe, PowerPoint, and Outlook).
- Minimum skills in MS Excel include creation of tables and Pivot tables, lookup functions, and use of basic statistical functions.
PREFERRED skills, experience, and professional characteristics
- Three or more years of experience in psychometrics or exam development, preferably in licensure or certification
- Experience training subject matter experts
- Experience implementing program evaluation
- Proficiency with statistical software (e.g., R, SPSS, WINSTEPS)
- Experience conducting research and communicating results through written and verbal communication
- Track record of remote work success
Employer Profile:
The American Board of Ophthalmology (ABO) serves patients and the public as the medical specialty board responsible for verifying the competencies of eye physicians and surgeons in the United States. A 501(c)(6) not-for-profit organization, the ABO certifies medical and osteopathic physicians who have completed an accredited ophthalmic residency training program and demonstrated the necessary level of knowledge, judgment, skills, and experience in ophthalmology. The ABO’s Continuing Certification program for board-certified ophthalmologists promotes career-long learning and ongoing improvement in patient care. The ABO is a founding member of the American Board of Medical Specialties (ABMS) and serves more than 20,000 active ophthalmologists. The organization is led by a physician-CEO and a 16-member Board of Directors; the ABO currently employs 11 full-time staff. Since 2018, the ABO has been recognized with GuideStar’s Platinum Rating for fiscal and operational transparency: https://www.guidestar.org/profile/23-1693176.
Work Environment:
This role is performed in a remote setting with all necessary work-at-home equipment provided. Required hours are 8:30 AM ET to 4:30 PM ET Monday through Friday, with a one-hour break for lunch, although flexibility in working schedule is supported with manager approval. Additional weekend hours may be required during examinations and special events (3-4 times per year). Domestic travel to in-person, all-staff meetings 2-3 times per year is required. These are frequently held in the Greater Philadelphia Area.
Base Salary:
$100,000-$120,000 per year, commensurate with experience and skills.
To Apply:
Please address resumes and letters of interest to Sarah Schnabel, PhD, Director of Psychometrics and Assessment.