TrueIQ.si
Submit a tool

PyRIT

microsoft.github.io · LLM Evals

Open source

Microsoft's open-source (MIT) Python Risk Identification Tool for red teaming generative AI, with composable targets, converters, scorers and multi-turn attacks.

#red-teaming#security#python#open-source
Visit PyRIT ↗GitHub ↗

About the LLM Evals category

Frameworks and platforms for testing language models and LLM applications — from academic benchmark harnesses to CI-friendly unit tests, LLM-as-judge scoring and red-teaming kits.

PyRIT is one of 19 llm evals tools indexed on TrueIQ. Facts on this page come from the tool's official site and public APIs; prices and features change, so confirm details on the official website before you commit.

// related

Related tools

All LLM Evals →
DeepTeamtrydeepteam.comOpen-source (Apache 2.0) LLM red teaming framework from Confident AI, built on DeepEval, covering 50+ vulnerability types with OWASP, NIST and MITRE presets.LLM EvalsOpen sourcegarakgarak.aiNVIDIA's open-source (Apache 2.0) LLM vulnerability scanner that probes models and endpoints for jailbreaks, prompt injection, data leakage and toxic output.LLM EvalsOpen source★ 9.4kInspect AIinspect.aisi.org.ukMIT-licensed framework from the UK AI Security Institute for rigorous LLM and agent evaluations, with tool use, sandboxed tasks and 200+ prebuilt evals.LLM EvalsOpen source★ 2.9kGiskardgiskard.aiOpen-source Python library for evaluating and red teaming AI agents and RAG apps. Version 3 is a modular rewrite for multi-turn testing; v2 is no longer maintained.LLM EvalsOpen source★ 5.9kPromptfoopromptfoo.devOpen-source CLI for testing prompts, models and RAG apps with assertions, side-by-side comparisons and automated red teaming. Part of OpenAI since 2026 and still MIT licensed.LLM EvalsOpen source★ 25.7kDeepEvaldeepeval.comPytest-style framework for unit-testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 18.6k