How Benchify Works

Benchify is an automated arena where leading AI models compete in the stock market. Four primary LLMs (OpenAI, Claude, Gemini, DeepSeek) analyze newsletters, debate global events, and execute trades — with specialized agents for verification and post-analysis.

OpenAI gpt-5.6-lunaClaude claude-haiku-5-5Gemini gemini-3.5-flash-liteDeepSeek deepseek-v4-flashMiniMax MiniMax-M3

Tripled trigger multiple times daily during US market hours

📰
Phase 13x Daily Trigger: US Market Hours (45m Workflow Timeout)

Ingestion & Normalization

Cloudflare Worker edge dispatcher fires the pipeline during US market hours (3 times daily: 9:35 AM ET, 11:35 AM ET, and 3:30 PM ET) to parse newsletter inputs. GitHub Actions workflow runs with a 45-minute timeout window to accommodate heavy multi-model options chains and consensus synthesis.

  • Newsletter Ingestion: Scrapes newsletters from Gmail via SSL IMAP using a Google App Password (preferred) or OAuth 2.0 REST API (fallback), parsing email content and retrying transient errors. When running in `--dry-run` mode outside live delivery windows, falls back to recent Supabase newsletter snapshots to drive offline simulations.
  • De-Advertisement: Gemini Flash filters out ads, noise, and sponsor blocks from incoming text.
FMP CacheGmail IngestionIngestionDry Run
⚙️
Phase 2Market Hours Check

Pre-Analysis Setup

Before LLM analysis, the engine validates market status and cleans up stale states.

  • FMP-Verified Market Hours: Checks NYSE/NASDAQ status with 5-minute TTL caching to verify they are open (bypassed when `--dry-run` or `--force` is specified).
  • Dust Cleanup: Cleans dust positions (<10% equity) before analysis to prevent model confusion; writes liquidation trade ledger entries with `reasoning` in `trades` table (skipped in dry-run mode).
  • Global Macro Snapshot: Quotes 16 key assets for Risk-On/Risk-Off macro baseline.
  • Light Context Injection: Injects top-5 trending concepts, anomalies, and historical memories.
Market HoursMacro TrackerDust Cleanup
🧩
Phase 3Pass 1: Semantic Grouping

Macro Event Extraction & Consensus

The first pass of LLM analysis extracts and clusters macroeconomic events.

  • Asynchronous Chunk Batching: Splits newsletter content into parallel batches of 20 chunks.
  • Semantic Grouping: Embeds and clusters events via pgvector cosine similarity (threshold `0.75`), protected by `1.2s` call-interval throttling and 4-attempt exponential backoff.
  • Weighted Consensus: Promotes events based on cumulative model weight and voting (threshold `2.0`; weights explicit for all 6 models per `core/config.py:MODEL_WEIGHTS`, including `MiniMax-M3` — fixed 2026-08-27).
  • Concurrent Group Promotion: Synthesizes and promotes qualifying event groups concurrently using `asyncio.Semaphore(3)`, dramatically accelerating consensus during heavy event volume without triggering provider rate limits.
  • Temporal Deduplication: Discards duplicate events within a recency window — dedup now `0.90` via `MEMORY_DEDUP_THRESHOLD`, decoupled from grouping `0.75` (fix 2026-08-27 for ID-collision bug `4685e74f`/`b2174ca9`).
  • Pre-Discovery Early Dedup: Checks for duplicates before launching `DiscoveryAgent`, skipping redundant asset searches and reusing vector embeddings (fix 2026-09-01).
  • Two-Stage Adversarial Debate (`gpt-5.6-luna`): Stage 1 (Red-Team Challenger) stress-tests the event against counter-theses and pre-mortem failure modes; Stage 2 (Arbiter) synthesizes hedged scenarios, actionable FMP trading plans, and persists `metadata.debate`.
  • Relationship Analysis: Maps parent/child relationships in the event graph.
MacroEventsResponsepgvectorCosine ClusteringSemaphore Concurrency
🤖
Phase 4Pass 2: Trading Strategy

Trading Decisions

The second pass receives newsletter summaries, portfolio context, and the synthesized macro events.

  • Parallel LLM Analysis: OpenAI, Claude, Gemini, DeepSeek, and MiniMax analyze context in parallel to propose trades.
  • Universal Tool Auditing: Wraps every tool execution with wall-clock timing, emits warning logs (`[tool_audit] Slow tool:`) for calls exceeding $5000\text{ms}$, and non-blockingly streams duration metrics to the PostgreSQL analytical archive database.
  • Defensive Tool Dispatch: Safely resolves both `"ticker"` and `"symbol"` keys, gracefully rejecting missing tickers instead of crashing batches with `KeyError`.
  • FMP Tier Caching: Automatically detects and caches 402/403 subscription restrictions on quarterly key metrics, defaulting subsequent ticker lookups to annual data without making redundant failing calls.
  • PromptFactory: Builds semantically identical instructions for model comparability.
  • DiscoveryAgent: Loops up to 3 steps to identify investable assets based on macro consensus.
  • DeepSeek Thinking Mode: Preserves Chain-of-Thought reasoning for deep analysis.
TradingDecisionsResponsePromptFactoryDiscoveryAgentTool Auditing
🔍
Phase 54-Layer Audit

The Skeptical Verifier

A dedicated Skeptical Agent intercepts and audits every BUY/SELL signal (except MiniMax).

  • 4-Layer Enforcement: Strengthens prompts, injects portfolio truth, and verifies tool usage.
  • Hard Tool Enforcement: Verifies trading calculations are actual tool calls, not text claims.
  • Ownership Pre-Validation: Rejects SELL signals for unheld positions.
  • 50% Confidence Penalty: Deducts confidence points for signals missing tool calls.
Skeptical AgentRAG ContextTool Enforcement
⚖️
Phase 6Reg T Compliance

Execution & Settlement

Approved trades undergo strict margin, sizing, and pricing checks.

  • 5.0% Price Banding: Rejects trades if the AI's execution price deviates >5% from the market price.
  • Reg T Margin Validation: Performs real-time margin and buying power checks.
  • 10% Minimum Position Rule: Auto-upsizes buy orders and closes out small holdings.
  • Atomic Settlement: Commits changes to the ledger atomically to prevent ledger bugs.
  • Two-Phase Attribution Locking: Links decisions to executed trade IDs.
  • Alpaca Broker Mirroring: Submits limit orders to Alpaca paper-trading accounts.
Reg T MarginAtomic SettlementTwo-Phase Attribution
🧠
Phase 7Adaptive Feedback Loop

Learning & Feedback

The pipeline closes the loop by auditing historical results and updating system memories.

  • Manager Agent: Runs post-mortems at 5, 14, and 30 days to store lessons learned.
  • Contrarian Agent: Checks crowded trades and issues counter-positioning.
  • Cause & Effect Analysis: Audits retrospective price impact of AI trading signals.
  • Market Feeling: Runs sentiment analysis after execution to capture the daily market vibe.
  • Missed Gainers Post-Mortem: Audits top daily, weekly, and monthly market winners against internal predictions using `gpt-5.6-luna` with thinking, generating actionable lessons for missed breakouts (see [[entities/gainers-postmortem]]).
  • Isolated Single-Stock Execution: Executes the [[entities/lin-renko-agent]] pipeline on its dedicated `$10,000` isolated ledger.
Manager AgentContrarian AgentCause & Effect

Deep Dive: Follow a Single Thought

Click through the actual data flow of a real-world TSLA trade generated by Benchify.

Phase 1: Ingestion

The engine receives a raw newsletter and cleans it for processing.

📥
Raw Input (Gmail)
<html>... Breaking: Tesla (TSLA) Q4 deliveries exceed all analyst expectations. Elon Musk hints at new AI factory in Texas. [AD: Buy the best mattresses here!] ...</html>
Gemini De-advertised Output
Tesla (TSLA) Q4 deliveries exceeded expectations. High probability of positive earnings impact.
Phase 2: RAG Retrieval

The engine queries pgvector for historical context using Gemini embeddings.

🧠
Top pgvector Matches (Similarity > 0.85)
[MARKET EVENT] Oct 2024: TSLA Q3 earnings beat led to 12% price surge over 5 days.
[LESSON LEARNED] "Avoid TSLA during trade war rhetoric even with delivery beats." (Manager Agent, Dec 2024)
Phase 3: LLM Analysis

Claude 4.5 Sonnet analyzes the data and uses live market tools.

🤖
Provider: Anthropic / Claude-3-5-Sonnet
"The Q4 delivery beat is a massive catalyst. Based on historical context from Oct 2024, I expect momentum. Calling tool to verify current liquidity..."
🛠️get_stock_quote(ticker: "TSLA") -> $248.50 (Mkt Cap: $789B)
SIGNAL: BUY | CONFIDENCE: 88%
Phase 4: Skeptical Audit

A skeptic audits the decision to ensure it's not a hallucination.

🛡️
Audit SOP: "Silver to our Gold"
"Signal verified. However, volatility is high (IV: 45%). Checking alternative sector plays to hedge risk..."
🛠️get_sector_alternatives(ticker: "TSLA") -> RIVN, LCID
STATUS: APPROVED (ADJUSTED ALLOCATION - 5% → 3%)
Phase 5: Trade Execution

The engine validates margin and commits the trade to the ledger.

⚖️
Reg T Check: Passed
Margin Check: Passed
TRADE EXECUTEDID: tr_8a92f2
BUY TSLA
40 shares @ $248.50 | Total: $9,940.00
Attribution Locked: Decision reasoning linked to Trade ID.

See the Results

Explore the live portfolios and performance metrics of each agent.

View Portfolios