Intelligence Analyst

Requirements

• Source evaluation and assessmentMust
• Confidence-level assessment and analytic standardsMust
• Exceptional written communication with clarity and precisionMust
• Exceptional verbal communication with clarity and precisionMust
• Ability to identify and articulate nuanced model limitations and over-refusal scenariosMust
• All-source analytic experience or deep expertise in a specific analytic discipline
• Civilian intelligence community (IC) background
• Structured analytic techniques
• Experience teaching, mentoring, or leading analytic reviews
• Record of fostering analytic rigor
• Previous participation in AI evaluation or model assessment initiatives

What you'll do

• Develop and map analytic cases to taxonomies across evaluation areas
• Author complex test cases to evaluate model performance on source handling and analytic rigor
• Assess and grade AI-generated responses based on reasoning, source use, and recognition of limitations
• Identify and flag instances of model over-refusal with documentation and feedback
• Document edge cases and analytic disagreements to support evaluation methodology improvement
• Collaborate with project leads to refine source evaluation and confidence-level assignment procedures
• Contribute insights to calibrate analytic standards and enhance AI system benchmark quality

Keywords

About the role

Role Title: Intelligence Analyst Role Type: Contractor Location: Remote micro1 is engaging Intelligence Analysts to contribute to a customer-facing project focused on evaluating and advancing artificial intelligence capabilities. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters. Scope of Work • Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios. • Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression. • Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility. • Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback. • Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies. • Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment. • Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks. Preferred Qualifications • Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background. • Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards. • Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor. • Exceptional written and verbal communication skills, with a focus on clarity and precision. • Ability to identify and articulate nuanced model limitations and over-refusal scenarios. • Familiarity with structured analytic techniques and intelligence tradecraft standards. • Previous participation in AI evaluation or model assessment initiatives is a plus. Originally posted on Himalayas