The week in brief
Behind it, Promptfoo (+54) and Laminar (+39) followed. This first comparison spans 1.5 days; from next week each report compares a full seven days.
One new tool joined the directory this week: Humanity's Last Exam (HLE).
On Hacker News, “Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers” drew the most attention in this space (165 points, 75 comments). Among listed tools, Artificial Analysis's site was linked from 87 Hacker News stories over the past 90 days, more than any other tool here.
In research, “AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents” collected the most upvotes on Hugging Face Daily Papers this week (23). Among new GitHub repositories created in the past 7 days, MohitBareja16/smriti leads with 7 stars.
Biggest GitHub star gainers
GitHub star counts for the 33 open source tools we track, compared between Oct 5, 2026 and Oct 7, 2026 (37 hours apart).
| Tool | Stars before | Stars after | Gain | Change |
|---|---|---|---|---|
| Langfuse | 35,410 | 35,480 | +70 | 0.20% |
| Promptfoo | 25,730 | 25,784 | +54 | 0.21% |
| Laminar | 3,321 | 3,360 | +39 | 1.17% |
| Opik | 22,391 | 22,425 | +34 | 0.15% |
| Aider Leaderboards | 49,384 | 49,411 | +27 | 0.05% |
| DeepEval | 18,649 | 18,675 | +26 | 0.14% |
| Arize Phoenix | 11,716 | 11,741 | +25 | 0.21% |
| Ragas | 15,937 | 15,951 | +14 | 0.09% |
New listings this week
| Tool | Category | Listed |
|---|---|---|
| Humanity's Last Exam (HLE) | Benchmarks | Oct 07, 2026 |
Most discussed on Hacker News
The highest scoring Hacker News stories in this space during the week.
Listed tools with the most Hacker News links
Hacker News stories linking to each tool's own website over the past 90 days.
Notable new papers
The most upvoted papers on Hugging Face Daily Papers and the newest relevant arXiv preprints from this week.
- AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents
- UNREAL: Unifying Retrieval and Long Context with a Single Model
- SEER: Self Evolving Event Reasoning and Retrieval for Time Series Forecasting
- DiVeR: Decision-Critical Verifier Learning for VLA Test-Time Scaling
- Judged Useless, Queried Anyway: Tool Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
- The Missing Minimal Pair: Stereotype Evaluation in LLMs
- Holdout Best-of-N: Unbiased Evaluation and Its Cost
- ScienceClaw: Benchmarking Continual Self Evolution of AI for Science Agents Across the Natural and Social Sciences
- Variance-Optimal Off Policy Evaluation with Conjunct Effect Modeling
- ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
New GitHub repos in LLM evaluation
Repositories created in the 7 days before Oct 07, 2026, ranked by stars.
| Repository | Stars | Language | Description |
|---|---|---|---|
| MohitBareja16/smriti | 7 | Python | Smriti (स्मृति): private, local first agentic RAG with System 1 + System 2 thinking over your notes, books and documents. Guardrails, tracing and evals included. |
| matank001/bossfight | 5 | Python | BOSSFIGHT: can a frontier LLM run a business? An end to end benchmark of AI business managers. |
| alymoursy/hieraticbench | 4 | TypeScript | Can AI read ancient Egyptian handwriting? A sealed benchmark for hieratic, with an open harness, dataset and leaderboard. |
| wxmb01/Claude-Detection | 4 | Python | 检测 Claude API 中转 / 代理是否掺水（模型替换、偷换、降智、注入、截断）的 Claude Code Skill 与参考实现 |
| SYasJ/llm-hallucination-detector | 2 | JavaScript | Free, open source LLM hallucination detector. See token level confidence as a heatmap and check every claim against your own sources (RAG). Works with any OpenAI compatible API, ChatGPT & Claude. Zero |
| read2017/ai-divination-benchmark | 2 | Python | 24 个普通人的 46 件人生大事，实测 12 款开源命理工具：八字 BaZi、紫微斗数 Zi Wei Dou Shu、印度占星 Vedic Astrology、奇门遁甲 Qi Men、六爻 Liu Yao、塔罗 Tarot。工具组第一名没超过「四句好话」对照，原始回答与评分可离线复核。 |
| mertkayacs/jevoss | 2 | Python | Evaluate Jev compatible decision models: accuracy, probability calibration, prompt injection, option order sensitivity and consistency. |
| SeCuReDmE-main-dev/orbit-kaggle-benchmark | 1 | Jupyter Notebook | Scoped evidence benchmark: native Boolean pilot, synthetic replay, separate C/C4 provenance and reproducibility |
How we compile this report
Every figure comes from a public source: star counts from the GitHub API, story scores and links from Hacker News (via Algolia), papers from arXiv and Hugging Face Daily Papers, and new repositories from GitHub search. Hacker News link counts include stories whose link points to the tool's own domain. Listing dates are the day a tool joined TrueIQ. Numbers are shown as recorded; if a figure is unavailable, it is left out rather than estimated.