Remote Agent Integration Protocol
A minimal protocol with two endpoints: /info for schema discovery and /invoke for task execution.
Back to Archive
01 / AI Evaluation
Protocol-Driven Platform for Agent-Agnostic Evaluation of LLM Agents
A research and engineering platform for evaluating heterogeneous LLM agents through a minimal integration protocol, reproducible benchmarks, and structured metric definitions.
AXI Technologies
Master thesis research internship
Master thesis research internshipFirst-author paperNeurIPS 2025 LLM Evaluation Workshop PosterSan Diego, USA · December 2025Master's Thesis · ExcellentIndustry-Lab Collaboration
At a glance
Project summary
Research Area
LLM Agent Evaluation · Evaluation Protocols · Benchmarking
Role
First Author · Platform Designer · Full-stack Implementation
Context
Master thesis internship at AXI Technologies, in collaboration with LIMOS
Output
NeurIPS 2025 LLM Evaluation Workshop Poster · Excellent thesis evaluation
System
Web-based evaluation platform for heterogeneous LLM agents
Core Protocol
Remote Agent Integration Protocol: /info and /invoke
Stack
Next.js · Django/DRF · PostgreSQL · Docker · FastAPI remote agent
Problem
Evaluation is still too coupled to implementation.
LLM-based agents are difficult to evaluate fairly because each system exposes different interfaces, output formats, and internal architectures. Existing evaluation workflows often require manual adaptation, making comparisons less reproducible and harder to scale.
Research contribution
Protocol first, agent internals second.
This project proposes a protocol-driven evaluation setup that separates agent evaluation from agent implementation. Instead of adapting the benchmark to each agent, agents expose a minimal common interface through the Remote Agent Integration Protocol.
Declarative task configuration
TaskConfig defines dynamic input templates and deterministic output extraction.
Agent-agnostic evaluation workflow
Heterogeneous agents can be evaluated under the same benchmark structure.
Metric taxonomy
11 metric types consolidated into 4 categories for structured evaluation.
Reproducible benchmark setup
Versioned benchmarks and locked configurations support consistent experiment runs.
System Architecture
Independent layers for invocation, orchestration, persistence, and scoring.
Benchmark
Orchestrator Core
Remote Agent via RAIP
Evaluation Plan
Persistence
Results
The platform separates benchmark definition, agent invocation, orchestration, persistence, and evaluation logic into independent layers.
Evaluation setup
Validated on a Course Advisor benchmark.
Task: Course AdvisorDataset: 30 examplesCompared setups: function-calling agent vs workflow-graph/ReAct-style agentEvaluation: metrics-based scoring and LLM-as-judge pipelineFocus: interoperability, reproducibility, output extraction, scoring consistency
Technical implementation
Research infrastructure presented as native web content.
Presentation Layer
Next.js 14 · React 18 · Zustand · Radix UI · Tailwind · Recharts · Monaco EditorDashboards, benchmark management, configuration editing, and result visualization.
Application Layer
Django · Django REST Framework · JWT · Pydantic · OpenAI/Anthropic integrationEvaluation orchestration, agent registry, benchmark execution, LLM-as-judge logic, and RAIP integration.
Data Layer
PostgreSQL · ORM models · Versioned benchmarks · JSON fieldsPersistence for agents, benchmarks, runs, metrics, results, and schema digests.
Infrastructure Layer
Docker · Docker Compose · Multi-service runtimeIsolated backend, database, and remote-agent services for reproducible execution.
Tooling & Automation
Justfile · Pytest · Playwright · Rouge-score · NLTKUnified commands for migration, seeding, evaluation, testing, and metrics execution.
Publication
A Protocol-Driven Platform for Agent-Agnostic Evaluation of LLM Agents
Cong Minh Tran, Issam Falih, Hatim Chahdi, Romain de la Souchere
NeurIPS 2025 LLM Evaluation Workshop · Poster · San Diego, USA · December 2025