AI directory search

Search across educators, skills, and resources.

Use this when you know the topic you need: Claude Code, MCP, evals, RAG, agents, product, coding, prompting, foundations, or model internals.

54 matches for "evals"

GPT-5.6 Claude Code Gemini 3.7 Flash Grok 4.6 MCP context engineering evals RAG OpenRouter coding agents

Video matches

Watch first when you want a fast feel for the topic before opening courses, docs, or profiles.

►

LLM evaluation with W&B

Weights & Biases · evals, llm apps, observability, mlops

►

AI evals with Phoenix

Arize AI · evals, observability, tracing, rag debugging

►

Promptfoo red teaming

Promptfoo · evals, prompt testing, red teaming, security

►

AI Evals for Engineers & PMs

Hamel Husain and Shreya Shankar · evals, product, llm reliability

Educators

Hamel Husain

Hamel's AI evals guides · Intermediate to advanced

Very practical material on evaluating LLM apps before they disappoint users.

Skills

Evals, RAG, LLM product quality

Shreya Shankar

AI Evals for Engineers and PMs · Intermediate

Useful if you need to judge whether an AI feature is actually improving.

Skills

Evals, LLM reliability, Product quality

Practical developer-focused AI education across LLM fundamentals, AI SDK app development, MCP, Claude Code workflows, agent-ready codebases, evals, TDD, handoffs, and reusable skills such as /teach, /grill-me, /to-prd, /to-issues, /tdd, /triage, and /handoff.

Skills

AI coding, Claude Skills, Agentic workflows, AI SDK, MCP, LLM fundamentals, Personalized learning

Agentic AI for Product Managers

Hamza Farooq on Maven · Beginner to intermediate

Useful for PMs who need to design, evaluate, and ship reliable AI systems beyond impressive demos.

Skills

Agentic AI, AI product strategy, Evals, Production AI

Providers and platforms

Weights & Biases

W&B Courses · Intermediate

Good for builders who need to measure, debug, and improve LLM apps rather than just demo them.

Topics

LLM apps, Evals, Experiment tracking, MLOps

OpenAI

OpenAI docs, Academy, and Cookbook · Beginner to advanced

Official model and implementation material for learning GPT-6 Astra and cost-sensitive GPT-5.6 choices, Codex workflows, subagents, memories, agent evals, MCP and connector patterns, retrieval, background jobs, prompt engineering, production best practices, model optimization, structured outputs, and OpenAI's Academy learning path.

Topics

GPT-6 Astra, GPT models, Reasoning models, Model selection, Agents, Subagents, RAG, Structured outputs, MCP, Evals, Memories

Arize AI

Phoenix · Intermediate

Useful for debugging and evaluating LLM applications once you move beyond prototypes.

Topics

Observability, Evals, Tracing, RAG debugging

Langfuse

Langfuse Docs · Intermediate

Good operational material for tracing, scoring, and improving production LLM apps.

Topics

Observability, Prompt management, Evals, Tracing

Vellum

Vellum Guides · Beginner to intermediate

Useful for product and ops teams that need practical LLM product concepts without getting lost in research.

Topics

Prompt management, Evals, Workflow design

Humanloop

Humanloop Blog and Docs · Intermediate

Useful for teams building repeatable AI product processes around prompts, datasets, and evaluations.

Topics

Prompt management, Evals, LLM workflows

Promptfoo

Promptfoo Docs · Intermediate

Very practical for regression testing prompts, model changes, and LLM outputs.

Topics

Prompt testing, Evals, Red teaming

Maven AI courses

Maven AI courses · Beginner to advanced

Useful discovery surface for live courses taught by practitioners across AI product, work, and engineering.

Topics

AI product, AI leadership, AI workflows, Evals

OpenRouter

OpenRouter docs · Beginner to intermediate

Useful for learning model comparison, latest-family aliases, routing, fallback behavior, agent construction, project-specific evals, and API-compatible experimentation across proprietary and open model families.

Topics

Model routing, Model comparison, Auto Router, Agent SDK, Coding-agent harnesses, GPT models, Claude models, Gemini, Llama, Mistral, DeepSeek, Qwen, API examples, Evaluation

Learning paths

Evals and reliability

AI product teams

Open path

Learn first

Test sets
Human review
Regression checks
Quality metrics

Good matches

Hamel Husain

Shreya Shankar

Chip Huyen

Matt Pocock

Open next

Resources

AI SDK v6 Crash Course

Workshop · Matt Pocock · Intermediate

You want a structured AI SDK v6 course that covers model choice, text and object generation, UI streams, agents, persistence, context engineering, evals, and advanced app patterns.

ai sdk, llm apps, agents, streaming, evals

The AI Engineer Roadmap

Free tutorial · Matt Pocock · Beginner to intermediate

You want a guided path through core AI concepts, model selection, the AI engineering mindset, evals, and techniques for improving LLM-powered apps.

ai engineering, model selection, evals, llm apps

LLM Evals

Guide · Hamel Husain · Intermediate

Your AI app needs quality checks before users see it.

evals, quality, llm apps

Evaluating AI Agents

Short course · DeepLearning.AI · Intermediate

You need to test, trace, and improve agent workflows instead of judging only single LLM responses.

agent evals, evals, agents, reliability, tracing

Building and Evaluating Advanced RAG Applications

Short course · DeepLearning.AI · Intermediate

You already know basic RAG and need better retrieval, evaluation, and production-quality patterns.

rag, evals, retrieval, llm apps, ai engineering

The Anatomy of Harness Engineering

Coding agent evaluation guide · Google Developers Blog · Intermediate to advanced

You want Google's September 9, 2026 guide to evaluating coding agents with small behavioral checks, outcome-based assertions, and batch runs that catch regressions without treating a single benchmark score as the whole story.

coding agents, evals, behavioral evaluations, regression testing, harness engineering

How Meta built safety into Muse

Agent security architecture guide · Meta AI · Intermediate to advanced

You want Meta's September 8, 2026 technical account of defense-in-depth for a long-running personal agent, including isolated runtime cells, credential surrogates, a separate permission authority, tainted-egress tracking, scoped approvals, browser controls, red teaming, and prompt-injection evals.

meta, muse, agent security, prompt injection, least privilege

Agentic models, measured on the injections that move money

Open security benchmark · Hugging Face Community · Intermediate to advanced

You want a reproducible September 5, 2026 benchmark for comparing how agentic models handle indirect prompt injection, with public data, a public harness, control runs, tool-call traces, and outcome metrics tied to unauthorized payment actions.

agents, prompt injection, agent security, evals, tool use

How GitHub makes AI coding more cost efficient

Coding agent evaluation guide · GitHub · Intermediate to advanced

You want GitHub's September 2, 2026 evidence for measuring coding-agent efficiency across the whole task, including selective output compression, preserving useful context, benchmark regressions, and controlled production experiments.

github copilot, coding agents, context engineering, evals, cost optimization

Project HydraFusion

Model orchestration research preview · GitHub · Intermediate to advanced

You want GitHub's September 4, 2026 technical explanation of runtime model orchestration for coding tasks, including plan decomposition, draft-critique-revise patterns, model cascading, evaluation design, and quality-versus-cost tradeoffs.

github copilot, model orchestration, model selection, coding agents, evals

Designing the hf CLI for agents

Agent tooling and evaluation guide · Hugging Face · Intermediate

You want a measured guide to agent-friendly CLI design, including structured output, retry-safe commands, independent grading, and benchmarks showing where higher-level tools reduce calls and token use.

hugging face, coding agents, hf cli, evals, codex

Hugging Face Skills

Open-source skills and tutorial · Hugging Face · Intermediate to advanced

You want reusable, open-source agent skills for training, fine-tuning, evaluating, quantizing, and publishing models with Codex, Claude Code, Gemini CLI, or another skills-compatible coding agent.

hugging face, agent skills, codex, claude code, model training

How to Evaluate AI Agents

Evaluation guide · Hugging Face Community · Intermediate

You need a practical evaluation plan built around representative tasks, controlled environments, observable traces, outcome and constraint metrics, repeated trials, and production failures turned into regression tests.

hugging face, agents, evals, evaluation harnesses, trajectory metrics

Anthropic Model Hardware Standard preview

Agent interoperability standard · Anthropic · Intermediate to advanced

You want Anthropic's August 27, 2026 research preview of a model-agnostic standard for connecting agents to lab and manufacturing hardware through programmable interfaces and protocols such as MCP, with safety evaluation built into the rollout.

anthropic, model hardware standard, agents, mcp, robotics

Agent and Model Evaluations in Gemini Enterprise Agent Platform

Evaluation guide · Google AI for Developers · Intermediate to advanced

You want Google's practical July 31, 2026 guide to running the same agent and model evaluations during development and on production traffic, with metrics for quality, safety, grounding, tool use, and trajectories.

google, gemini, agents, evals, model selection

Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Training guide · Hugging Face · Advanced

You need an end-to-end August 26, 2026 recipe for adapting a late-interaction retriever to your own domain, from datasets and losses through training, evaluation, and index optimization.

hugging face, sentence transformers, rag, retrieval, fine-tuning

Automating repetitive work at OpenAI with Codex

Practical guide · OpenAI · Intermediate

You want an August 25, 2026 field guide to capturing recurring engineering and evaluation work in reviewable notebooks, keeping approval boundaries explicit, and feeding lessons from earlier agent runs into the next run.

openai, codex, automation, evals, runbooks

How to Choose the Best AI Model (Live, in Your Editor)

Model selection guide · OpenRouter · Intermediate

You want a practical six-step workflow for shortlisting models from live data, testing them on your own prompts, measuring cost per completed task, and choosing or routing from inside a coding assistant.

openrouter, model selection, evals, benchmarks, cost

OpenAI eval design guide

Guide · OpenAI · Intermediate

You need practical guidance for designing representative eval datasets, choosing graders, and turning model testing into an engineering loop instead of ad hoc spot checks, especially while OpenAI's older Evals platform is winding down toward read-only status on October 31, 2026.

openai, evals, quality, datasets, regression testing

OpenAI evals quickstart and datasets

Guide · OpenAI · Intermediate

You want OpenAI's current quickstart for turning examples into dataset-backed evals and improvement loops instead of relying on a deprecated docs path.

openai, datasets, evals, fine-tuning, quality

OpenAI deprecations

Migration guide · OpenAI · Intermediate

You need OpenAI's current retirement schedule and recommended replacements before pinning model IDs in evals, coding agents, or user-facing features.

openai, deprecations, migrations, model selection, release notes

OpenAI all models

Model catalog · OpenAI · Beginner to advanced

You want the current full OpenAI model catalog, including older and deprecated entries, before wiring exact model IDs into evals, tooling, or migrations.

openai, model catalog, gpt, codex, reasoning models

OpenAI Working with evals

Guide · OpenAI · Intermediate

You need API-level guidance for testing outputs, comparing models, and catching regressions during upgrades.

openai, evals, quality, regression testing, reliability

OpenAI Evaluate agent workflows

Guide · OpenAI · Intermediate

You need the current OpenAI path for tracing, grading, and regression-testing agent workflows instead of only single-prompt evals.

openai, agents, evals, traces, graders

OpenAI model optimization

Guide · OpenAI · Intermediate

You need a practical optimization loop across prompt changes, evals, and fine-tuning rather than guessing which knob to turn next.

openai, prompting, evals, fine-tuning, optimization

Claude model deprecations

Migration guide · Anthropic · Intermediate

You need Anthropic's official retirement and replacement schedule before pinning Claude model IDs in production, evals, or coding-agent defaults.

anthropic, claude, deprecations, migrations, model selection

Perplexity Search Evals

GitHub repo · Perplexity · Intermediate

You want Perplexity's open evaluation suites for benchmarking grounded search quality and comparing search API behavior against other retrieval stacks.

perplexity, search evals, benchmarks, grounded answers, evaluation

OpenAI model selection cookbook

Cookbook guide · OpenAI · Intermediate

You want a practical OpenAI walkthrough for model selection tradeoffs, eval design, and rollout testing instead of treating model choice as a static table lookup.

openai, model selection, evals, latency, cost

OpenRouter Ori Eval

Evaluation guide · OpenRouter · Intermediate to advanced

You want to turn real project prompts and data into repeatable cross-provider agent evals that check answers, tool calls, completion, latency, and cost, then rerun them in CI when models change.

openrouter, evals, model selection, coding agents, ci