AI Evaluations for Everyone: How Non-Engineers Can Build Better AI Systems with Humane Intelligence

Course details

Non-engineers can build better AI systems through humane intelligence. In this course, get introduced to practical domain-driven methods for evaluating AI systems against organizational missions, values, and real-world needs. Learn how to conduct human-in-the-loop assessments that test AI systems in meaningful contexts like public health and education. Discover how to identify potential misalignments and biases in AI outputs early in the development cycle. Observe lively discussions and review real-world exercises designed to bring abstract AI concepts into tangible evaluations. Whether you work on mission-driven teams, lead projects, or are part of policy operations, this course empowers you to make informed decisions about AI adoption. Gain skills in designing taxonomies and ontologies, running AI evaluations, and mitigating risks.

Audience:
Professionals responsible for AI adoption decisions
Mission‑driven teams
Organizations deploying AI for social impact or public interest
Product managers
Program leads
Policy teams
Content or operations professionals

Concepts

Introduction

You can build better AI systems

Building Your First AI Evaluation - Red Teaming

Why AI evals are important
Red teaming and other evals in practice
An AI red teaming workshop end-to-end
Introduction to the mini project (pizza shop red teaming exercise)

AI Contextual Evaluations

Mapping problem spaces - Taxonomical vs. ontological knowledge graph approaches
Mini project - Select and justify your evaluation strategy

Designing Taxonomies and Ontologies

The evaluation spectrum - From fully automated to fully human
Taxonomies and ontologies in practice
Mini project - Setting up your taxonomy

Identifying AI Failure Modes

Bias in data and models
Factuality and hallucinations
Misdirection and over-reliance
Security and adversarial risks
Mini project - Identify priority risks for fake pizza restaurant reviews

Running the Evaluation

Prompt selection and scenario design (governance)
Red teaming best practices
Expert review and annotation in red teaming
Mini project - Draft your red teaming plan

Final Project - From Findings to Decision

Making sense of red teaming data
Now what Working with red team results
Mitigation strategies and guardrails
Mini project - Select the best AI model and finalize your evaluation
Additional resources and next steps

AI Evaluations for Everyone: How Non-Engineers Can Build Better AI Systems with Humane Intelligence

Authors

Humane Intelligence

Mala Kumar

Annie Brown

Theodora Skeadas