TrueIQ.si
Submit a tool

EVALS · BENCHMARKS · ASSESSMENT

Measure what matters.

The directory for measurement in the age of AI: eval frameworks, benchmarks, leaderboards, observability and red teaming for machines, plus assessment and psychometrics for people.

Tools indexed
87
GitHub stars
338.9k
Categories
6
Open source
25
Free to start
74
Feed items
39
LIVE

// guides

Latest guides

All guides →
Explainers · Nov 20, 2026 · 8 min readAgent Arena Index: How Arena Ranks AI AgentsThe Agent Arena Index explained: the four signals behind Arena's agent leaderboard, how the score is weighted and anchored, and how to read the November 2026 board.Read the guide →ListenExplainers · Nov 19, 2026 · 10 min readGemini 4 Neon Benchmarks: Which Claims Hold UpGemini 4 Neon benchmarks checked against Artificial Analysis, Vals, Arena, Terminal-Bench, Zapier and Cognition boards: which launch claims match and which do not.Read the guide →ListenScreenshots of Helicone, LangSmith, Langfuse, RagasComparisonsComparisons · Nov 15, 2026 · 10 min readHelicone vs LangSmith: Proxy Logs or Agent Traces?Helicone vs LangSmith in 2026: request logs and a mature gateway against agent traces, evals and a new beta gateway. What each bills, sees and costs, tested.Read the guide →ListenSO-Arena Benchmark: How System One Models RankLeaderboardsLeaderboards · Nov 14, 2026 · 9 min readSO-Arena Benchmark: How System One Models RankSO-Arena tests System One decision models on 2,919 tasks in 11 domains. Who leads, why the top scorer is only seventh on reliability, and how to read it.Read the guide →ListenScreenshots of HoneyHive, Langfuse, LangSmith, BraintrustComparisonsComparisons · Nov 14, 2026 · 9 min readHoneyHive vs Langfuse: Free Tiers, Evals and HostingHoneyHive vs Langfuse in 2026: what each one bills as an event, free tier math from a real SDK test, evals, self hosting and which team should pick which.Read the guide →ListenScreenshots of GPTZero, QuillBot, RAID, PangramComparisonsComparisons · Nov 14, 2026 · 9 min readGPTZero vs QuillBot: Which Catches Paraphrased AI?GPTZero vs QuillBot in 2026: free limits, prices, our read of both RAID result files, and which detector caught QuillBot paraphrased AI text in outside tests.Read the guide →ListenScreenshots of GPTZero, ZeroGPT, RAID, PangramComparisonsComparisons · Nov 13, 2026 · 9 min readGPTZero vs ZeroGPT: Which AI Detector to Trust?GPTZero vs ZeroGPT in 2026: who makes each, free limits, prices, and what RAID and four independent studies found about false flags on real human writing.Read the guide →ListenExplainers · Nov 13, 2026 · 9 min readGemini 4 Hydrogen Benchmarks: What Holds UpGemini 4 Hydrogen benchmarks checked against Artificial Analysis, Vals, Terminal-Bench, Snorkel and Arena boards: which launch scores hold up and which do not.Read the guide →ListenScreenshots of ARC-AGI, Humanity's Last Exam (HLE), SEAL Leaderboards, GPQAComparisonsComparisons · Nov 13, 2026 · 10 min readARC-AGI vs Humanity's Last Exam: Key DifferencesARC-AGI vs Humanity's Last Exam: one tests how fast a model learns a new puzzle, the other tests expert knowledge. Scores, grading, cost and which to trust.Read the guide →Listen

// start here

Guides by topic

// field guide

A practical guide to measuring intelligence: artificial and human

You cannot improve what you do not measure, and in AI measurement has become its own discipline. TrueIQ is a directory for the tools that do that measuring. On one side are evaluation frameworks, benchmarks and observability platforms that tell builders whether a language model or agent is accurate, safe and worth its cost. On the other are assessment and psychometric platforms that measure human skills, aptitude and cognition. Both ask the same question: how do we know how capable something really is?

If you build with LLMs, start with the evals and benchmarks categories. Benchmarks such as MMLU, GPQA, SWE-bench and ARC-AGI are the shared yardsticks the field reports against. Academic harnesses such as EleutherAI's lm-evaluation-harness, Stanford's HELM and Hugging Face's lighteval run standard benchmarks reproducibly. Application level tools like Promptfoo, DeepEval, Ragas and Braintrust let you write tests for your own prompts and pipelines, run them in CI and catch regressions before users do. Leaderboards (Arena, Artificial Analysis, LiveBench, SEAL and others) help you shortlist models, but each measures something specific, from human preference to contamination resistant accuracy to tokens per second, so compare like with like.

Once an application is live, observability takes over. Platforms such as Langfuse, LangSmith, Arize Phoenix, Helicone and Opik trace every call, record latency and cost, and attach evaluation scores to real traffic. Good evaluation is continuous: the best teams feed production failures back into their test sets, and red teaming tools such as garak and PyRIT probe for jailbreaks and unsafe behaviour before attackers do. We also list integrity tools that estimate whether writing was AI generated; they are useful signals for educators and publishers but can produce false positives, so they should inform judgement rather than replace it.

The human side of TrueIQ covers assessment and psychometrics: classroom grading, technical hiring tests, aptitude and IQ style tests and research grade cognitive batteries. These tools are not medical devices, and no online test captures the whole of a person's intelligence; we describe what each product does, link to its official site and leave clinical claims to qualified professionals. The live feed below brings in new benchmark and evaluation papers from arXiv and Hugging Face, plus discussion from Hacker News.

// at a glance

The TrueIQ signal

What is rising, where the stars are, and what just landed.

Pricing mix

  • Open source25
  • Free34
  • Freemium15
  • Paid9
  • See site3

Fresh from the feed

    Open the live feed →

    Sources

    4 public sources

    Hacker News, arXiv, HF Daily Papers, GitHub. Every star count, link and paper points to its original source.

    Newest listing: SO-Arena

    // this week · week 47

    This week in numbers

    All weekly reports →
    Promptfoo adds 341 GitHub stars to lead the weekPromptfoo added 341 GitHub stars between Nov 9 and Nov 16, the biggest gain among the 42 open source tools tracked on TrueIQ.Read the week 47 report →
    • +341GitHub stars for Promptfoo
    • 147points on the week's top HN story
    • 10notable new papers

    // fresh

    New this week on GitHub

    // index

    The directory

    87 of 87 tools

    Artificial Analysisartificialanalysis.aiIndependent benchmarking of model intelligence, speed and price across providers, updated as new models launch.LeaderboardsFreeHN 91Langfuselangfuse.comOpen source LLM engineering platform (MIT core) for tracing, prompt management, datasets and evals, free to self host. Owned by ClickHouse since January 2026.ObservabilityOpen source★ 37.7kTerminal-Benchtbench.aiHarbor and Laude Institute agent benchmark of hard Linux terminal tasks. Version 4.0 has 66 tasks with an 8 hour timeout; 4.1 hardens verifiers and adds a 63 task CPU only set.BenchmarksFree★ 1.1kVals AIvals.aiIndependent evaluation lab that runs models on private, expert written tests in tax, law, finance, medicine and coding, with cost and time per test on every board.LeaderboardsFreeHN 28Laminarlaminar.shOpen source observability platform for AI agents with tracing, evaluations and browser session replays.ObservabilityOpen source★ 3.5kInspect AIinspect.aisi.org.ukMIT licensed framework from the UK AI Security Institute for rigorous LLM and agent evaluations, with tool use, sandboxed tasks and 200+ prebuilt evals.LLM EvalsOpen source★ 3.3kDeepEvaldeepeval.comPytest style framework for unit testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 19.6kJev (TypeSafe AI)typesafe.aiDecision model from TypeSafe AI that returns typed answers with probabilities instead of text. Used as a cheap judge in evals and guardrails.LLM EvalsPaidHN 16SWE-benchswebench.comOriginal Princeton SWE-bench of real GitHub issues where a fix counts only if the repo's tests pass; includes Lite, Verified and related subsets. For Scale AI's harder Pro benchmark, see SWE-Bench Pro.BenchmarksFree★ 6.2kPromptfoopromptfoo.devOpen source CLI for testing prompts, models and RAG apps with assertions, side by side comparisons and automated red teaming. Part of OpenAI since 2026 and still MIT licensed.LLM EvalsOpen source★ 27.5kOpikcomet.comComet's open source platform for logging, evaluating and monitoring LLM apps, RAG systems and agents.ObservabilityOpen source★ 23.5kRagasragas.ioOpen source library of RAG and LLM app evaluation metrics with synthetic test set generation, maintained by Vibrant Labs, the company founded by the creators of Ragas.LLM EvalsOpen source★ 16.4kArize Phoenixarize.comFree, self hostable tracing and evaluation tool built on OpenTelemetry, source available under the Elastic License 2.0. Arize became part of Dynatrace in October 2026.ObservabilityFree★ 12.4kAider Leaderboardsaider.chatAider's code editing benchmarks that test how well models edit real source files across several programming languages.LeaderboardsFree★ 50.2kgarakgarak.aiNVIDIA's open source (Apache 2.0) LLM vulnerability scanner that probes models and endpoints for jailbreaks, prompt injection, data leakage and toxic output.LLM EvalsOpen source★ 10.1kHumanity's Last Exam (HLE)lastexam.ai2,500 expert written questions from CAIS and Scale AI, plus HLE Diamond, a cleaned 1,000 question subset (500 reasoning, 500 knowledge) released September 2026.BenchmarksFree★ 1.7kLiveBenchlivebench.aiBenchmark refreshed with new questions regularly and scored against objective answers to limit test set contamination.LeaderboardsOpen source★ 1.4kOpenCompassopencompass.org.cnComprehensive evaluation platform from Shanghai AI Lab supporting many models, datasets and a public leaderboard.LLM EvalsOpen source★ 7.6kOpenLLMetrytraceloop.comOpenTelemetry based instrumentation for LLM providers and frameworks, sending traces to any observability backend.ObservabilityOpen source★ 7.5kHeliconehelicone.aiOpen source LLM gateway and request logger for tracking cost, latency and usage. Acquired by Mintlify in March 2026; the hosted service now runs in maintenance mode.ObservabilityOpen source★ 6.3kGiskardgiskard.aiOpen source Python library for evaluating and red teaming AI agents and RAG apps. Version 3 is a modular rewrite for multiturn testing; v2 is no longer maintained.LLM EvalsOpen source★ 6kPyRITmicrosoft.github.ioMicrosoft's open source (MIT) Python Risk Identification Tool for red teaming generative AI, with composable targets, converters, scorers and multiturn attacks.LLM EvalsOpen source★ 5kOpenAI Evalsgithub.comFramework and open registry of evaluations for testing language models and the systems built on them.LLM EvalsOpen source★ 19.8kTruLenstrulens.orgInstrumentation and feedback functions for evaluating and tracking LLM apps, known for the RAG triad metrics.LLM EvalsOpen source★ 3.7kMTEBhuggingface.coMassive Text Embedding Benchmark and leaderboard for comparing embedding models on retrieval, clustering and more.LeaderboardsOpen source★ 3.5klm-evaluation-harnessgithub.comEleutherAI's unified harness for running hundreds of academic benchmarks against any model with consistent prompts.LLM EvalsOpen source★ 14.5kVectara Hallucination Leaderboardgithub.comPublic leaderboard that scores how often models introduce unsupported facts when summarizing documents, using Vectara's open HHEM evaluation model.LeaderboardsFree★ 3.3kBerkeley Function Calling Leaderboardgorilla.cs.berkeley.eduLeaderboard measuring how accurately models call functions and tools across languages and multiturn settings.LeaderboardsFree★ 13.1klightevalgithub.comHugging Face toolkit for evaluating LLMs across many backends with reusable tasks and detailed sample level results.LLM EvalsOpen source★ 2.6kEvidentlyevidentlyai.comOpen source library for evaluating, testing and monitoring ML and LLM systems, from data drift to response quality.LLM EvalsOpen source★ 8.1kARC-AGIarcprize.orgAbstraction and Reasoning Corpus series from Francois Chollet and the ARC Prize Foundation that tests skill acquisition on novel tasks; ARC-AGI-3 adds interactive games.BenchmarksFree★ 4.9kW&B Weavewandb.aiWeights & Biases toolkit for tracing, evaluating and iterating on LLM applications.ObservabilityFreemium★ 1.2ksimple-evalsgithub.comOpenAI's lightweight reference code for zero shot benchmark runs. No longer updated for new models since July 2025; it now mainly hosts HealthBench, BrowseComp and SimpleQA.LLM EvalsOpen source★ 4.7kPangrampangram.comAI detector for text, images and video from Pangram Labs, with free daily allowances, heatmaps on Image 1.0 and paid API rates.Integrity & DetectionFreemiumHN 5BIG-benchgithub.comBeyond the Imitation Game: a collaborative benchmark of 200+ diverse tasks from 450 authors at 132 institutions, built to probe and extrapolate language model capabilities.BenchmarksFree★ 3.2kDeepTeamtrydeepteam.comOpen source (Apache 2.0) LLM red teaming framework from Confident AI, built on DeepEval, covering 50+ vulnerability types with OWASP, NIST and MITRE presets.LLM EvalsOpen source★ 3.1kHELMcrfm.stanford.eduStanford's Holistic Evaluation of Language Models, measuring accuracy, robustness, bias and efficiency across many scenarios.LLM EvalsOpen source★ 3kMMLUgithub.comMassive Multitask Language Understanding: a multiple choice benchmark spanning 57 subjects from math to law, introduced in 2020 and now largely saturated by frontier models.BenchmarksFree★ 1.6kSWE-Bench Prolabs.scale.comScale AI coding agent benchmark of long horizon GitHub style tasks with public, private and held out splits; V2 refreshes the public set to 642 validated tasks plus a HARD-51 subset.BenchmarksFree★ 569Arena (LMArena)arena.aiCrowdsourced arena, formerly LMArena, where people compare anonymous model responses side by side, producing preference based leaderboards.LeaderboardsFreeHN 3GPQAgithub.comGraduate level, Google proof science Q&A: 448 expert written biology, physics and chemistry questions, with a 198 question Diamond subset widely reported in model launches.BenchmarksFree★ 551MMLU-Progithub.comHarder successor to MMLU from TIGER-Lab: about 12,000 reasoning heavy questions in 14 disciplines with ten answer options, published at NeurIPS 2024.BenchmarksFree★ 435GDPxgdpx.semianalysis.comSemiAnalysis benchmark that runs AI agents on 206 long tasks in 11 domains such as chip design, finance and cybersecurity, with cost per task and mostly private tasks.LeaderboardsFreeHN 2SO-Arenalichenlillc.github.ioAcademic benchmark of 2,919 tasks in 11 domains that scores System One decision models on capability, speed and a separate reliability test for shuffled or missing options.BenchmarksOpen source★ 27Jev Decision Indexhuggingface.coCommunity Hugging Face leaderboard that runs open copies of TypeSafe's Jev, and Jev itself, on 37 public benchmarks and ranks them on private tests.LeaderboardsFree★ 100Infinity-Chathuggingface.coOpen dataset from the Artificial Hivemind paper: 26,070 real open ended chatbot questions plus 31,250 human ratings, used to measure how alike model answers are.BenchmarksFree★ 79NoveltyBenchnovelty-bench.github.ioCarnegie Mellon benchmark for output diversity: 1,100 prompts, each answered 10 times, scored on how many answers are meaningfully different and how useful the new ones are.BenchmarksFree★ 35Fingerprinting Leaderboardfingerprint.sentient.foundationOpen Sentient Foundation leaderboard that ranks LLM fingerprinting schemes on retention and adversarial robustness under a fixed detect neighbor attack.LeaderboardsFreeHN 1HackerRankhackerrank.comTechnical hiring platform with coding assessments, interviews and AI assisted evaluation of developer skills.Assessment & PsychometricsPaidHN 1mlprintsgithub.comOpen source AGPL-3.0 framework from Sentient for implementing, attacking and verifying LLM fingerprinting methods in a shared evaluation environment.LLM EvalsOpen source★ 3BenchLeaderbenchleader.comLeaderboard aggregator that combines published results from 45 benchmark publishers into one daily index and shows how far apart the sources disagree.LeaderboardsFreeBrainHQbrainhq.comAdaptive brain training exercises designed around research on processing speed, attention and memory.Assessment & PsychometricsPaidBraintrustbraintrust.devEnd to end platform for evaluating AI products, with datasets, scorers, prompt playgrounds and production logging.LLM EvalsFreemiumCambridge Cognitioncambridgecognition.comDeveloper of CANTAB digital cognitive assessments used in clinical research and trials.Assessment & PsychometricsPaidCodilitycodility.comSkills assessment platform for evaluating engineering candidates through coding tests and live interviews.Assessment & PsychometricsPaidContent Credentials Verifycontentcredentials.orgFree web tool that reads C2PA Content Credentials in a file and shows its signed provenance history, including whether AI tools were used to create or edit it.Integrity & DetectionFreeCopyleakscopyleaks.comAI text and image detector bundled with a plagiarism checker in one report, with three sensitivity levels, 30+ detection languages, LMS plugins and a credit based plan.Integrity & DetectionFreemiumCriteriacriteriacorp.comTalent assessment provider offering validated aptitude, personality and emotional intelligence tests for hiring.Assessment & PsychometricsPaidDatadog Agent Observabilitydatadoghq.comDatadog's hosted tracing and evaluation product for LLM apps and agents (formerly LLM Observability), billed per LLM call and linked to Datadog APM and infrastructure data.ObservabilityFreemiumEpoch AI Benchmarking Hubepoch.aiEpoch AI's independent runs of key benchmarks with open data showing how frontier capabilities change over time.LeaderboardsFreeFireworks Specialized Intelligence Indexfireworks.aiFree Fireworks AI leaderboards that score models on domain benchmarks built by companies such as Harvey, Sierra and Surge AI, with native metrics, cost and time per task and no overall score.LeaderboardsFreeFrontierCodecognition.comCognition's private coding agent benchmark that grades whether maintainers of real open source repositories would merge an agent's change, scoring scope, tests and code quality as well as correctness.BenchmarksSee siteGalileogalileo.aiEvaluation, observability and guardrail platform for AI agents and LLM apps. Cisco completed its acquisition in May 2026 and is folding it into Splunk Observability.LLM EvalsFreemiumGDPvalopenai.comOpenAI benchmark of real professional work: 1,320 expert written tasks across 44 occupations, with a public gold set of 220 tasks graded against human expert deliverables.BenchmarksFreeGPTZerogptzero.meAI text detector used by educators to estimate whether writing was produced by language models, with writing reports.Integrity & DetectionFreemiumGradescopegradescope.comGrading platform that speeds up marking of paper and online assessments with AI assisted answer grouping.Assessment & PsychometricsPaidHarness Leaderboardhuggingface.coHugging Face Space that runs nine coding agent harnesses on the same open models and tasks, with three attempts, confidence intervals, cost and public trajectories.LeaderboardsFreeHive AI Content Detectionthehive.aiHive (renamed Deploy on Oct 26, 2026) sells deepfake and AI generated content detection APIs that score images, video and audio, name the likely generator, and come with a free Chrome extension.Integrity & DetectionSee siteHoneyHivehoneyhive.aiPlatform for testing, monitoring and evaluating AI agents across development and production.ObservabilitySee siteKahoot!kahoot.comGame based quiz platform for classrooms and teams, with AI tools for generating quizzes from content.Assessment & PsychometricsFreemiumLangSmithlangchain.comLangChain's platform for tracing, evaluating and monitoring LLM apps and agents, framework agnostic.ObservabilityFreemiumLumositylumosity.comCognitive game app with daily training sessions that adapt in difficulty and track performance over time.Assessment & PsychometricsFreemiumMKA (Multimodal Knowledge Agent)huggingface.coGoogle DeepMind agent benchmark of 200 professional tasks where key facts sit in drawings, schematics, maps and charts, graded on weighted rubrics and run publicly by Artificial Analysis.BenchmarksFreeOpenAI Content Verificationopenai.comOpenAI's free checker and Content Provenance API that look for its SynthID watermarks and C2PA data in images and audio made with ChatGPT, Codex or the API.Integrity & DetectionFreeOriginality.aioriginality.aiAI content and plagiarism checker aimed at publishers and content teams.Integrity & DetectionPaidPatronus AIpatronus.aiResearch lab for training data and simulated environments for frontier AI. Its former LLM evaluation platform, evaluators, Lynx, GLIDER and Percival are retired; new users cannot get API keys (see docs.patronus.ai).LLM EvalsContact onlyQuillBotquillbot.comWriting suite best known for its paraphraser, with a free AI detector that sorts text into AI, AI refined and human bands, bundled with a humanizer in Premium.Integrity & DetectionFreemiumRAIDraid-bench.xyzOpen benchmark and leaderboard for AI text detectors from the University of Pennsylvania: over 6 million texts across 11 generators, 8 domains and 11 attacks, scored at a fixed false positive rate.BenchmarksFreeReality Defenderrealitydefender.comEnterprise deepfake detection for audio, video and images across live calls, meetings and uploaded files, delivered through an API and integrations.Integrity & DetectionFreemiumSEAL Leaderboardslabs.scale.comScale AI's expert driven leaderboards using private datasets to reduce contamination and gaming.LeaderboardsFreeSynthIDdeepmind.googleGoogle DeepMind's invisible watermark for AI images, video, audio and text. Files can be checked in the SynthID Detector portal, open to all since October 2026, or in Gemini.Integrity & DetectionFreeTestGorillatestgorilla.comPre employment testing library covering cognitive ability, personality and role specific skills.Assessment & PsychometricsFreemiumTurnitinturnitin.comLong standing academic integrity platform offering similarity checking, AI writing indicators and feedback tools for schools.Integrity & DetectionPaidVellum Leaderboardvellum.aiComparison table of recent models across reasoning, coding and cost metrics, maintained by the Vellum platform.LeaderboardsFreeVIALS-AAartificialanalysis.aiArtificial Analysis's independent board for Handshake AI's VIALS test. It scores 161 scientific image tasks as pass cubed across three repeats.BenchmarksFreeWinston AIgowinston.aiAI text and image detector with a 0 to 100 Human Score, sentence maps, plagiarism checks on upper plans, website certification and a credit based API.Integrity & DetectionFreemiumZeroGPTzerogpt.comFree AI text detector with a 15,000 character paste box and sentence highlights, sold alongside a humanizer, paraphraser and other writing tools, with paid plans and a prepaid API.Integrity & DetectionFreemium

    // taxonomy

    Categories

    // live feed

    Research & news feed

    Sources: Hacker News, arXiv, HF Daily Papers, GitHub

      // submit

      Submit a tool

      Know a tool that belongs in TrueIQ? Send it in. Every submission is reviewed by a human; listings are never sold.

      // newsletter

      Get The .si AI Tools Weekly

      A short weekly digest, every Monday: the best new tools and research in Evals, curated from the live feed above, plus top picks from our sister .si directories.

      Free, weekly, every Monday. Delivered by Beehiiv; you confirm by email. Unsubscribe any time. Privacy