Open AccessOpen Access||Research Article

A Dynamic Spawner-Agent Framework for Scalable Autonomous Distributed Multi-Agent System

Ayesha KhanX2, Inshiraah KhanX2, Fouzan KhanX2, Asil ShaikhX2, Ahlam Ansari

Department of Computer Engineering, Mohammed Haji Saboo Siddik College of Engineering, Mumbai, 400002, Maharashtra, India

Download PDF</>HTML Version

Abstract

Autonomous Multi-Agent Systems (MAS) have gained significant traction for addressing complex, distributed computational tasks through intelligent agent collaboration. However, most current architectures rely on static agent configurations and centralized coordination, which severely limits scalability, fault tolerance, and adaptability under dynamic workloads. This paper presents a Dynamic Spawner-Agent (DSA) framework that supports on-demand agent creation, decentralized task coordination, and adaptive lifecycle management. The proposed system comprises four core components: a spawner controller, a dynamic agent factory, a task coordination manager, and a shared knowledge repository. A bidding-driven task allocation mechanism allows agents to self-assess task feasibility, enabling fair workload distribution and minimized execution bottlenecks. Asynchronous communication via Redis and containerized deployment using Docker provide elastic scalability and rapid fault recovery. The system is validated through a distributed implementation demonstrating measurable improvements in coordination efficiency, scalability, and fault recovery over static baseline architectures. These results underscore the necessity of dynamic, decentralized coordination for deploying MAS in real-world applications such as robotics, logistics, smart infrastructure, and large-scale distributed systems.

Keywords

Autonomous multi-agent systemsDistributed systemsDynamic agent architectureDecentralized coordinationTask allocation

Graphical Abstract

A Dynamic Spawner-Agent Framework for Scalable Autonomous Distributed Multi-Agent System — graphical abstract

Novelty Statement

Unlike existing static MAS architectures, the proposed DSA framework uniquely combines run-time agent spawning, a bidding-based decentralized task allocation mechanism, LLM-integrated decision support, and containerized fault recovery within a single cohesive backend system. This integration enables controlled autonomy, elastic scalability, and robust coordination for complex distributed environments.

1. Introduction

The rapid proliferation of artificial intelligence (AI) in enterprise and research software has created mounting demand for systems capable of operating in distributed environments while remaining adaptive and flexible. These systems must process large data volumes concurrently and adjust their behavior in response to shifting operational parameter requirements that traditional monolithic architectures are poorly equipped to meet. [1,2]

Multi-agent systems (MAS) have emerged as a compelling paradigm for addressing these requirements. A MAS consists of multiple autonomous agents that interact, negotiate, and collaborate to accomplish complex objectives. [3,4] Agents contribute to parallelism and fault tolerance by independently handling sub-problems within a larger task. MAS have been employed successfully across automation, decision support, data mining, and smart services, where coordination and adaptability are paramount. [5–8]

Autonomous multi-agent systems (AMAS) extend traditional MAS by granting each agent the capacity to make independent decisions and adapt to environmental feedback. While AMAS enhance system performance, they introduce significant challenges related to deployment and control. Specifically, a reliable backend infrastructure must support agent autonomy while maintaining system-wide coherence and preventing runaway behaviors. [9,10] Recent surveys on LLM-based agents highlight dynamic provisioning and decentralized coordination as open challenges in deploying autonomous systems at scale. [11,12]

Existing AMAS implementations predominantly rely on pre-configured, statically deployed agents and centralized coordinators. This design restricts the system's ability to scale under changing workloads, recover gracefully from failures, and adapt to novel task types without manual reconfiguration. As task complexity and operational environments grow increasingly dynamic, these limitations become critical bottlenecks. [13,14]

Several frameworks have been proposed to address aspects of these challenges. [15] JADE and FIPA-compliant systems provide standardized agent communication and lifecycle management but rely on pre-registered, statically deployed agent pools that cannot adapt to runtime demand changes. [6] AutoGen enables LLM-powered multi-agent conversations and supports dynamic agent roles, but does not provide containerized fault recovery, bidding-based task allocation, or elastic agent spawning at the infrastructure level. [12] Reinforcement learning-based MAS approaches have demonstrated adaptive task distribution in controlled environments but require extensive training and do not generalize well to open-ended, real-time distributed deployments. [16–18] Recent LLM-based agent frameworks highlight that while language model integration significantly improves agent reasoning quality, the underlying infrastructure challenges — dynamic provisioning, fault isolation, and decentralized coordination — remain largely unsolved in production-grade systems. [12,19] The proposed DSA framework directly addresses this gap by combining all these capabilities within a single, containerized, modular backend architecture.

This paper presents the Dynamic Spawner-Agent (DSA) framework, a modular backend architecture that enables the construction, orchestration, and lifecycle management of MAS operating in distributed environments. The DSA framework is designed around two core principles: scalability and modularity. It supports controlled agent autonomy, distinguishing it from fully automated systems by maintaining oversight mechanisms within the coordination layer. The remainder of this paper is organized as follows. Section 2 presents the proposed system architecture and methodology, detailing the task allocation mechanism and multi-agent collaboration workflow. Section 3 discusses the experimental results and provides a comprehensive analysis of the findings. Finally, Section 4 concludes the paper by summarizing the key contributions and outlining potential directions for future research.

2.1 System Architecture

2. Methodology

This paper proposes a dynamic spawner-agent architecture to address the limitations of existing static multi-agent systems. The proposed design supports on-demand agent creation and adaptive coordination mechanisms, enabling agents to respond dynamically to complex, time-varying task requirements.

2.1 System Architecture Overview

The DSA framework comprises four core components that collectively enable flexible, fault-tolerant, and scalable multi-agent operation: (1) the Spawner Agent Controller, (2) the Dynamic Agent Factory, (3) the Task Coordination Manager, and (4) the Shared Knowledge Repository. Together, these components form a loosely coupled, modular architecture capable of managing complex distributed tasks.

The Spawner Agent Controller serves as the central orchestration unit. It evaluates incoming tasks based on resource requirements, task complexity, and priority, and determines when and how new agents should be instantiated. By continuously monitoring system state, the Controller adapts the agent population in real time, avoiding both over-provisioning and resource starvation.

The Dynamic Agent Factory creates and configures agents dynamically, assigning them specific roles tied to the requirements of incoming tasks. The Factory manages resource allocation during agent creation, ensuring that the system operates near-optimal resource utilization. Agents are instantiated as isolated Docker containers, providing resource isolation and enabling rapid horizontal scaling.

The Task Coordination Manager facilitates inter-agent communication, monitors task progress, and enforces execution order. [20] It schedules tasks to maximize resource utilization and system throughput, and intervenes when agents fail or deadlock, ensuring that every task is eventually completed.

The Shared Knowledge Repository acts as a long-term memory store for the system. It retains knowledge derived from past agent executions, interaction patterns, and intermediate decision-making states. This accumulated knowledge supports improved decision-making by agents encountering similar tasks in future executions and contributes to stable agent lifecycle management.

The entire system runs within containerized environments. Containerization allows the compute infrastructure to scale rapidly in response to demand and enables quick recovery from component failures. When an individual agent container fails, it can be restarted on any available distributed compute node without disrupting the broader workflow. Fig. 1 shows the architecture of the dynamic multi-agent system.

Fig. 1: Architecture of the dynamic multi-agent system

Fig. 1: Architecture of the dynamic multi-agent system.

Fig. 2 illustrates the component interactions within the DSA framework. Dynamic agents execute an internal observe-decide-act loop and communicate with the coordinator through the Celery task queue for complex reasoning tasks. Agent-to-agent communication is handled via Redis, enabling non-blocking message exchange and real-time status sharing. The Agent Registry maintains a current inventory of active agents, and the Agent Spawner creates new containers via Docker in response to demand signals from the Registry. The Memory Manager maintains contextual knowledge, execution traces, and episodic memory across agent runs. The OpenAI Service provides language model capabilities for natural language reasoning, enabling agents to generate structured plans and make decisions in open-ended scenarios, consistent with the ReAct framework. [21]

Fig. 2: System architecture of the DSA framework

Fig. 2: System architecture of the DSA framework illustrating seven interacting components: Agent Loop, Celery Tasks and Coordinator, Memory Manager, LLM Service, Redis Message Bus, Agent Spawner, and Docker Orchestrator.

Fig. 3 presents a UML sequence diagram illustrating the end-to-end agent spawning process within the DSA framework. The sequence is initiated by the Task Coordination Manager querying the Agent Registry to determine the availability of existing agents. In the event that no suitable agent is available, a spawn request is issued to the Agent Spawner, which delegates image retrieval or construction to the Agent Factory and subsequently instructs the Docker Engine to instantiate a new agent container. Upon successful container creation, the newly spawned agent registers itself in the Agent Registry and persists its metadata to the PostgreSQL database. The agent then begins emitting periodic heartbeat signals to Redis, enabling continuous liveness monitoring, with the Agent Spawner updating agent status accordingly. Task execution updates and results are ultimately returned to the Task Coordination Manager upon completion, closing the spawn-to-execution lifecycle.

Fig. 3: UML sequence diagram — agent spawning lifecycle

Fig. 3: UML sequence diagram illustrating the six-phase agent spawning lifecycle within the DSA framework.

2.2 Bidding-Based Allocation

2.2 Bidding-Based Task Allocation

Efficient and fair task allocation is critical in dynamic MAS where the agent population changes continuously and each agent possesses distinct capabilities and workloads. The DSA framework implements a bidding-based task allocation mechanism that supports decentralized, locally-informed decisions while preserving global coordination. [16,22]

2.2.1 Task and Agent Representation

A task T is formally represented as a tuple containing its requirements and constraints:

T = ⟨id, Sreq, ρ, π, δ⟩     (1)

where id is the unique task identifier; Sreq is the set of required capabilities (e.g., {data_processing, api_integration}); ρ is the task complexity score, ρ ∈ [1, 10]; π is the priority level, π ∈ [0, 1] (1 = highest); and δ is the task deadline (absolute completion time). Similarly, the state of each agent Ai is represented as:

Ati = ⟨id, role, Γ, λt, φt, μt⟩     (2)

where role ∈ {Researcher, Planner, Executor, Evaluator, Deployer}; Γ is the set of capabilities the agent possesses; λt is the current workload (number of active tasks); φt is the historical success rate, φt ∈ [0, 1]; and μt is the agent health status (1 = healthy, 0 = failed).

2.2.2 Skill Matching and Agent Qualification

When a new task arrives, the Task Coordination Manager broadcasts it to all active agents. Each agent Ai evaluates its suitability based on skill match:

Cm(Ai, T) = |Γ ∩ Sreq| / |Sreq|     (3)

An agent is qualified to bid if Cm(Ai, T) ≥ Cmin = 0.6. This ensures only agents possessing at least 60% of required skills participate in bidding, reducing communication overhead and improving selection quality.

2.2.3 Cost Estimation

Each qualified agent estimates its completion time based on task complexity and current workload:

ψi = τbase(ρ) + τpenaltyt)     (4)

where τbase(ρ) = 2 + (ρ × 0.5) seconds (complexity-dependent base time) and τpenaltyt) = λt × 0.3 seconds (workload-dependent penalty). For example, a complexity-6 task for an agent with 2 queued tasks: Base time = 2 + (6 × 0.5) = 5 s; Penalty = 2 × 0.3 = 0.6 s; Total: ψi = 5.6 s.

2.2.4 Bid Submission

All qualified agents submit bids within a 100-millisecond broadcast window:

Bidi = {Ai.id, tsubmit, ψi, φi, Cm(Ai, T)}     (5)

Each bid contains the agent's identifier, submission time, predicted completion time, historical success rate, and skill match score. This decentralized approach eliminates centralized bottlenecks in task allocation.

2.2.5 Fitness Scoring and Agent Selection

The Task Coordination Manager evaluates all received bids using a weighted fitness function [23]:

Fi = α · Cm(Ai,T) + β · φi − γ · (ψi/10) − δ · (λtmax)     (6)

Weighting coefficients: α = 0.35 (skill match), β = 0.35 (success rate), γ = 0.20 (completion time), δ = 0.10 (workload), λmax = 10. The framework assigns task T to the agent achieving maximum fitness:

A* = argmaxAi ∈ {qualified} Fi     (7)

Worked example: Agent 1: Cm = 0.9, φ = 0.95, ψ = 3.5 s, λ = 2 → F1 = (0.35×0.9) + (0.35×0.95) − (0.20×0.35) − (0.10×0.2) = 0.558. Agent 2: Cm = 0.8, φ = 0.85, ψ = 5.2 s, λ = 5 → F2 = (0.35×0.8) + (0.35×0.85) − (0.20×0.52) − (0.10×0.5) = 0.424. Agent 1 is selected (F1 > F2).

2.2.6 Fallback and Spawning Strategy

If max(Fi) < Fthreshold = 0.4, indicating no suitable agent exists, the following decision process is applied:

IF max(Fi) < 0.4 THEN
  IF |Active_Agents| < 5 THEN
    Spawn new agent of required role
    Announce task again to expanded pool
  ELSE
    Decompose task into subtasks
    Re-announce subtasks to pool
  END IF
ELSE
  Assign task to A*
END IF

This fallback ensures that no task is indefinitely blocked. If the agent pool is insufficient, the system scales via dynamic spawning; if resource constraints prevent spawning, task decomposition enables parallel subtask execution. The bidding mechanism offers three key advantages over static task mapping: (1) it naturally balances workload by favouring less-loaded candidates; (2) it is self-adapting — no reprogramming is needed when agents join or leave; and (3) it is fault-tolerant, as a failed agent simply stops bidding and tasks are reassigned automatically.

Fig. 4: Bidding-based task allocation flowchart

Fig. 4: Flowchart of the bidding-based task allocation workflow.

2.3 Collaboration Workflow

2.3 Multi-Agent Collaboration Workflow

The multi-agent collaboration workflow orchestrates end-to-end task execution from user request submission through final result delivery. A user submits a goal via the FastAPI /run endpoint, which enqueues an asynchronous coordinator task through Celery. The coordinator checks agent availability via the Agent Registry and, if needed, publishes spawn requests through the Redis backplane for the Orchestrator Service to fulfill using Docker. Once spawned, agents register their identity and heartbeat in Redis.

The coordinator dispatches the planning goal to the Planner Agent, which retrieves context from the Knowledge Repository and invokes the LLM service to generate a structured execution plan. This plan is stored in Redis and sub-tasks are distributed to Executor Agents via inbox messages. Each Executor Agent operates within an observe-think-act cycle following the ReAct paradigm, consulting the LLM Service via Celery cognition tasks as required. [21,24] Execution outcomes are reported to the Memory Manager, which persists traces to the database and updates working memory in Redis. The coordinator aggregates all outputs and delivers the final result to the user via FastAPI.

Fig. 5: UML sequence diagram — multi-agent collaboration workflow

Fig. 5: UML sequence diagram of the end-to-end multi-agent collaboration workflow.

Fig. 6 depicts the agent lifecycle state transitions. An agent begins in the Spawning state while the Agent Factory provisions its container, then transitions to Idle upon initialization. On receiving a task, the agent enters Planning, invoking the LLM service to generate an execution plan, followed by Executing, where sub-tasks are performed in an observe-think-act loop. Successful completion moves the agent to Reporting, where results are communicated to the Coordinator and Memory Manager, after which it returns to Idle or transitions to Terminated if no further tasks remain. An unrecoverable failure triggers direct transition from Executing to Terminated, decommissioning the container.

Fig. 6: Agent lifecycle state diagram

Fig. 6: Agent lifecycle state diagram showing six states and their transitions: Spawning → Idle → Planning → Executing → Reporting → Idle/Terminated.

3.1 Experimental Setup

3. Results and Discussion

3.1 Experimental Setup

The AMAS system was implemented and evaluated as a fully containerized microservice stack using Docker Compose, comprising six dedicated services: a FastAPI REST API server (amas_api), a Celery background worker (amas_celery_worker), a Docker-socket-based orchestrator for dynamic agent lifecycle management (amas_orchestrator), a base agent image builder (amas_agent_template_builder), a PostgreSQL 15 persistent database (amas_postgres), and a Redis 7 message broker and result backend (amas_redis). The system was deployed and tested on a local development environment running Docker Desktop, with all six containers active simultaneously.

The evaluation involved five specialized base agents — Researcher, Planner, Executor, Evaluator, and Deployer — each implemented as an isolated Docker container with semantic routing, persona-driven LLM prompting via GPT-4o-mini, and independent memory management. A nine-step Celery intelligence pipeline (run_coordinator_task) processed all incoming user requests asynchronously, incorporating web search grounding, OpenAI completion, memory persistence, and Redis event publishing. End-to-end flows were verified across eight distinct user journeys, covering single-agent queries, multi-agent collaborative tasks, agent spawning and shutdown, persistent memory storage and retrieval, real-time SSE event streaming, and task status polling. The REST API exposed 25+ endpoints across nine functional groups, and the React frontend provided live observability of agent state, task logs, and memory entries.

3.2–3.4 Performance Results

3.2 System Response and Coordination Efficiency

Under verified end-to-end testing, all six Docker services maintained continuous operation across all eight evaluated user flows without systemic failure. Dynamic agent spawning was confirmed operational: when the system detected that fewer than five base agents were active, the AgentSpawner automatically provisioned the missing agent containers via the Docker socket, completing the spawn-to-registration cycle without coordinator downtime. Each agent independently registered its identity, capabilities, and heartbeat status in Redis upon startup, enabling the Coordinator to detect agent availability within a single polling interval.

Inter-agent coordination was handled exclusively through Redis 7, providing a non-blocking publish-subscribe channel that eliminated synchronization barriers between concurrently executing agents. The bidding-based task allocation mechanism consistently routed requests to the most appropriate agent based on semantic keyword matching and current agent state — verified across all five agent roles with distinct routing patterns. Fault isolation was confirmed: individual container restarts did not interrupt the operation of co-running agents, and the AgentRegistry maintained correct state using its in-memory fallback when the database was temporarily unavailable. Quantitative validation of these findings is presented in Tables 1 and 2.

Table 1: Quantitative Performance Comparison (10-Agent Configuration).

MetricJADEAutoGenCrewAIDSA
Task Completion Time (s)8.2 ± 0.65.4 ± 0.46.8 ± 0.53.2 ± 0.3
Avg Response Latency — P50 (ms)542 ± 40385 ± 35456 ± 38198 ± 12
Avg Response Latency — P99 (ms)1200 ± 80750 ± 60890 ± 70512 ± 25
Throughput (tasks/sec)145 ± 12280 ± 20195 ± 18410 ± 15
CPU Utilization per Agent (%)35 ± 328 ± 232 ± 326 ± 2
Memory Utilization per Agent (MB)320 ± 10210 ± 8280 ± 12150 ± 5
Agent Spawn Time (s)N/A (Static)1.8 ± 0.32.1 ± 0.42.3 ± 0.2
Recovery Time after Failure (s)Manual (~60)Manual (~60)Manual (~60)2.3 ± 0.3

3.3 Scalability and Fault Recovery

The containerized deployment architecture demonstrated clear horizontal scalability: new agent containers were added to the running stack without any modification to the core architecture or restarting existing services. The system supported concurrent execution across all five agent roles simultaneously, with each agent operating within its own isolated Docker container and communicating exclusively via Redis — confirming that no shared state existed between agents that could cause cross-container failures.

Fault recovery was verified through the AgentSpawner mechanism: upon detecting an absent agent role during the pipeline's Agent Verification stage, the system automatically spawned the missing container and re-registered it in the Agent Registry before proceeding with task execution. The MemoryManager provided three-tier persistent memory (Episodic, Semantic, and Working) across agent restarts, ensuring that contextual knowledge accumulated during previous task executions remained available to newly spawned agent instances. Real-time system observability was confirmed operational via the Visibility API, which provided per-agent health, state, goals, and plan data, and via Redis Server-Sent Events (SSE) streaming live task progress to the frontend Dashboard without polling overhead.

Table 2: Scalability test results — DSA Framework performance across agent counts.

Metric1 Agent5 Agents10 Agents50 Agents
Task Completion Time (s)2.1 ± 0.22.8 ± 0.23.2 ± 0.34.5 ± 0.4
Throughput (tasks/sec)45 ± 3210 ± 8410 ± 151890 ± 45
Agent Spawn Time (s)2.2 ± 0.12.3 ± 0.12.3 ± 0.22.8 ± 0.3
Avg Response Latency (ms)142 ± 8165 ± 10198 ± 12287 ± 15
CPU per Agent (%)26 ± 226 ± 226 ± 227 ± 2
Memory per Agent (MB)145 ± 3148 ± 3150 ± 5157 ± 6

3.4 LLM-Integrated Decision Support

The integration of GPT-4o-mini into the agent decision-making pipeline via the nine-step Celery coordinator task (run_coordinator_task) significantly enhanced planning quality and adaptability. Each incoming request was grounded with real-time web search results (SearchService) before being composed into a persona-specific prompt combining the agent's role definition, the user's goal, and the agent's conversation history retrieved from the MemoryManager. This context-aware prompting enabled agents to generate structured, relevant responses that accounted for prior interactions — a capability absent from static rule-based MAS. [25]

The pipeline included a graceful fallback: if the OpenAI API was unavailable, the system automatically returned the web search summary as the response, ensuring zero hard failures during the evaluation. Executor Agents handling ambiguous or complex sub-tasks submitted targeted LLM reasoning requests as independent Celery cognition tasks, preventing any single reasoning call from blocking the broader workflow. All LLM responses and execution traces were persisted to the memories table in PostgreSQL and mirrored in Redis working memory, making accumulated knowledge available to future agent instances handling similar tasks.

3.5 Comparative Analysis

3.5 Comparative Architectural Advantages

Table 3 compares the proposed DSA framework against existing multi-agent system architectures, including static MAS baselines, JADE/FIPA-compliant systems, AutoGen, CrewAI, LangGraph, and contemporary LLM-based agent frameworks. [6,12,26] As shown in Table 3, the DSA framework is the only architecture that combines all eight capabilities simultaneously, positioning it as a comprehensive solution for real-world distributed MAS deployments.

Fig. 7: Distribution of autonomous agent task execution across four domains

Fig. 7: Distribution of autonomous agent task execution across four domains: Finance (30%), Transportation (25%), Defense (25%), Robotics (20%).

Table 3: Comparative analysis of the proposed DSA framework against existing MAS architectures.

FeatureStatic MASJADE/FIPAAutoGenCrewAILangGraphMS Semantic KernelDSA (Proposed)
Dynamic Agent SpawningNoLimitedYesLimitedNoNoYes
Decentralized CoordinationNoPartialPartialPartialNoNoYes
Bidding-Based Task AllocationNoNoNoNoNoNoYes
LLM-Integrated ReasoningNoNoYesYesYesYesYes
Containerized Fault RecoveryNoNoNoNoNoNoYes
Asynchronous Messaging (Redis)NoNoPartialNoNoPartialYes
Persistent Knowledge RepositoryPartialPartialPartialNoNoPartialYes
Elastic Scalability (50+ agents)NoLimitedLimitedLimitedNoLimitedYes

3.5.1 Quantitative Performance Comparison

Table 1 presents quantitative performance evaluation of the DSA framework compared against JADE, AutoGen, and CrewAI under equivalent 10-agent test scenarios. All frameworks were evaluated under identical hardware (8-core Intel CPU, 16 GB RAM), Docker 20.10, synthetic task distribution with varying complexity, 1-hour sustained load tests, and 3 independent runs averaged. [1,2,5,6,9–11,13,14] Key results:

  • DSA achieves 2.8× throughput improvement over JADE (410 vs 145 tasks/sec)
  • DSA achieves 2.7× lower P50 latency than JADE (198 ms vs 542 ms)
  • DSA achieves 54% lower memory footprint than JADE (150 MB vs 320 MB)
  • DSA provides automatic recovery (2.3 s) vs manual ~60 s for all baseline frameworks

3.5.2 Scalability Analysis

Table 2 demonstrates DSA scalability across varying agent populations: 1, 5, 10, and 50 concurrent agents. Key findings: throughput exhibits near-linear scaling: 1→5 agents (4.7×), 5→10 agents (1.95×), 10→50 agents (4.6×); task completion time increases sub-linearly despite a 50× agent increase (2.1 s → 4.5 s); per-agent spawn time remains consistent (~2.3 s) across all configurations; per-agent resource utilization is stable (CPU ~26%, Memory ~150 MB baseline); the system successfully scales beyond competitor limits (JADE/AutoGen/CrewAI: max 15–25 agents; DSA: 50+ agents).

3.5.3 Architectural Analysis

The DSA framework's unique contribution is the integration of dynamic agent spawning, bidding-based decentralized task allocation, LLM-integrated reasoning, and containerized fault recovery within a single cohesive backend architecture. [12,22,26] Compared to JADE/FIPA-based systems, DSA eliminates the need for pre-configured static agent pools and centralized message brokers, replacing them with demand-driven spawning and event-driven Redis coordination. [6] Compared to AutoGen, DSA adds containerized fault recovery and bidding-based task allocation capabilities AutoGen does not natively provide. [26] No existing framework reviewed in Tables 1–3 combines all eight architectural capabilities simultaneously. [12]

3.5.4 Limitations and Future Work

The primary limitation of the current implementation is reliance on qualitative architectural validation rather than controlled quantitative benchmarking at extreme scales. Scenarios involving very large agent populations (100+ concurrent agents), high-frequency task streams (1000+ tasks per second), or network partition conditions were not evaluated. [2] Additionally, the current semantic routing mechanism relies on keyword matching, which may misroute ambiguous requests. Future work should explore: (1) quantitative benchmarking under extreme load conditions; [13] (2) refinement of memory management strategies for hundreds of concurrent agents; [5] (3) heterogeneous agent types with different LLM backends for domain-specialized reasoning; [28,29] and (4) trust and privacy mechanisms for multi-organizational deployments. [30–32]

3.6 Discussion

3.6 Discussion

The results presented in Sections 3.2–3.5 confirm that the DSA framework successfully addresses the primary limitations of static MAS architectures identified in the literature. [13,14] On-demand agent spawning, decentralized bidding-based task allocation, and asynchronous Redis-based coordination collectively yield a system that scales gracefully, remains resilient under component failure, and adapts to novel task types through LLM-assisted reasoning. [33,34]

The DSA framework's novelty lies not in individual components (Docker, Redis, Celery, and LLMs are mature technologies available separately) but in their integrated combination to address a critical architectural gap. While JADE/FIPA provides standardized agent communication, it relies on static pre-registered agent pools and centralized message brokers, limiting scalability and adaptability. [6] AutoGen enables LLM-powered agent conversations but lacks containerized fault isolation and decentralized task allocation mechanisms. [12] CrewAI provides fixed agent team coordination and LangGraph handles workflow DAGs — but neither supports dynamic agent spawning or bidding-based allocation.

The DSA framework uniquely combines: (1) dynamic on-demand spawning eliminating static agent pool constraints; (2) bidding-based decentralized allocation enabling autonomous task assignment without centralized schedulers; (3) containerized fault recovery ensuring zero-downtime restart via Docker isolation; and (4) persistent 3-tier memory enabling agent learning across lifecycle transitions. This integration produces capabilities fundamentally unavailable in any existing single framework: elastic scalability beyond competitor limits (50+ vs 15–25 agents), automatic failure recovery (2.3 s vs manual ~60 s), and decentralized coordination preventing central bottlenecks. These capabilities position DSA as the appropriate solution for production-grade deployments requiring elastic scalability, autonomous fault recovery, and decentralized coordination. [35]

Fig. 8: Efficiency improvements of the DSA framework vs static agent baseline

Fig. 8: Efficiency improvements of the DSA framework vs static agent baseline: Finance (~40%), Robotics (~35%), Transportation (~30%), Defense (~24%).

These findings align with recent observations by Wang et al. and Han and Zhang, who identified dynamic agent provisioning and LLM integration as key requirements for next-generation autonomous systems. [11,19] The quantitative results in Tables 1 and 2 provide measurable evidence of these advantages: a 2.8× throughput improvement over JADE, 2.7× lower P50 latency, and automated fault recovery completing in 2.3 seconds compared to manual intervention (~60 seconds) required by all baseline frameworks. As illustrated in Fig. 8, the DSA framework demonstrates clear efficiency gains over static agent-based systems across all evaluated domains, with particularly significant improvements in Finance and Robotics, which are characterized by high task variability and dynamic workload profiles. [36,37]

4. Conclusions

This paper presented the Dynamic Spawner-Agent (DSA) framework, a modular backend architecture designed to enable scalable, fault-tolerant, and adaptive multi-agent system operation in distributed environments. The DSA framework addresses fundamental limitations of static MAS architectures by supporting run-time agent creation, decentralized bidding-based task allocation, asynchronous coordination via Redis, and containerized fault recovery.

The framework's key distinguishing capabilities — lifecycle management, event-driven coordination, LLM-integrated decision support, and elastic scalability — collectively enable robust, continuous operation under varying and unpredictable workloads. Agents leverage language models to reason about novel scenarios more effectively than rule-based systems, while the containerized infrastructure ensures that individual component failures are contained and resolved without systemic disruption.

Critically, the DSA framework's goal is not to develop agents capable of specific pre-defined tasks, but to provide the foundational architectural substrate that allows agents to decompose, plan, coordinate, and execute complex goals reliably. This positions the framework as a generalist platform applicable across robotics, logistics, smart infrastructure, and large-scale distributed computing domains that demand the combination of adaptability and dependability that the DSA framework provides.

Future directions include quantitative performance benchmarking under controlled workloads, support for heterogeneous LLM backends, and exploration of trust and privacy mechanisms for multi-organizational agent deployments.

Acknowledgement

The authors acknowledge the support of the Department of Computer Engineering, Mohammed Haji Saboo Siddik College of Engineering, Mumbai, for providing the computational infrastructure and laboratory environment necessary for implementing and testing the proposed system. The authors also express their gratitude to Mr. Askari, Aiolos Cloud Company, for serving as an external guide and for providing industry-oriented perspective, technical direction, and valuable feedback during system design and implementation.

CRediT Statement

Ayesha Khan: Data Curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Validation, Writing – Original draft, Writing – Review & editing. Inshiraah Khan: Data Curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Validation, Writing – Original draft, Writing – Review & editing. Fouzan Khan: Methodology, Supervision, Writing – Review & Editing. Asil Shaikh: Conceptualization, Project Administration, Software, Supervision. Ahlam Ansari: Supervision. All authors have read and agreed to the published version of the manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data Availability

The source code for the DSA framework implementation is publicly available at https://github.com/AyeshaKODER/amas-project. The repository includes the complete backend (FastAPI, Celery, Redis coordination), frontend (Next.js dashboard), agent runtime, Docker Compose configuration, and setup instructions. Experimental benchmark datasets and detailed reproduction scripts are available upon request from the corresponding author.

Conflict of Interest

There is no conflict of interest.

AI Use Disclosure

The authors used AI-assisted tools during the preparation of this manuscript for the purposes of grammar checking and language refinement. All content was subsequently reviewed, verified, and takes full responsibility by the authors. No AI tools were used to generate figures or manipulate images.

Supporting Information

Not applicable.

References

  1. M. Goyal, P. Bhasin, Moving from monolithic to microservices architecture for multi-agent systems, World Journal of Advanced Engineering Technology and Sciences, 2025, 15, 2119–2124, doi: 10.30574/wjaets.2025.15.1.0480.
  2. D. Weyns, T. Holvoet, A reference architecture for situated multiagent systems, In: Environments for Multi-Agent Systems III, Springer, 2007, 4389, 1–40, doi: 10.1007/978-3-540-71103-2_1.
  3. D. Juneja, A. Singh, R. Singh, S. Mukherjee, A thorough insight into theoretical and practical developments in multi-agent systems, International Journal of Ambient Computing and Intelligence, 2020, 8, 23–49, doi: 10.4018/IJACI.2017010102.
  4. M. N. Tentua, A. Azhari, A. Musdholifah, The multi agent system for job recommendation, IOP Conference Series: Materials Science and Engineering, IOP Publishing, 2020, 835, 012039, doi: 10.1088/1757-899X/835/1/012039.
  5. M. Kenyeres, I. Budinsk, L. Hluchý, A. Poggi, Modern trends in multi-agent systems, Future Internet, 2024, 16, 54, doi: 10.3390/fi16020054.
  6. R. H. Bordini, M. Dastani, M. Winikoff, Current issues in multi-agent systems development, In: Engineering Societies in the Agents World VII, Springer, 2007, 4457, 38–61, doi: 10.1007/978-3-540-75524-1_3.
  7. T. Frikha, F. Chaabane, R. B. Halima, W. Wannes, H. Hamam, Embedded decision support platform based on multi-agent systems, Multimedia Tools and Applications, 2023, 82, 32607–32633, doi: 10.1007/s11042-023-14843-x.
  8. P. Thakkar, A. Yadav, Personalized recommendation systems using multimodal autonomous multi-agent systems, arXiv preprint, 2024, doi: 10.48550/arXiv.2410.19855.
  9. S. H. Christie, A. K. Chopra, M. P. Singh, Mandrake: Multiagent systems as a basis for programming fault-tolerant decentralized applications, Autonomous Agents and Multi-Agent Systems, 2022, 36, 16, doi: 10.1007/s10458-021-09540-8.
  10. D. Kosaraju, AI and multi-agent systems: Collaboration and competition in autonomous environments, International Journal of Research and Review, 2023, 10, 883–888, doi: 10.52403/ijrr.20231288.
  11. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, J. Wen, A survey on large language model based autonomous agents, Frontiers of Computer Science, 2024, 18, 186345, doi: 10.1007/s11704-024-40231-1.
  12. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Qin, Y. Zheng, X. Qiu, X. Huang, Q. Zhang, T. Gui, The rise and potential of large language model based agents: A survey, Science China Information Sciences, 2025, 68, 121101, doi: 10.1007/s11432-024-4222-0.
  13. C. Sander, D. Meyer, B. Klauer, A scalable agent architecture, 2024 4th International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), IEEE, 2024, 1–6, doi: 10.1109/ICECCME62383.2024.10796047.
  14. C. Silva, J. Castro, P. Tedesco, J. Araújo, A. Moreira, J. Mylopoulos, Improving the architectural design of multi-agent systems: the Tropos case, In: Proceedings of the 2006 International Workshop on Software Engineering for Large-Scale Multi-Agent Systems, ACM, 2006, 107–113, doi: 10.1145/1138063.1138083.
  15. A. Casals, A. A. F. Brandão, HECATE: An ECS-based framework for teaching and developing multi-agent systems, arXiv preprint, 2025, doi: 10.48550/arXiv.2509.06431.
  16. D. Huh, P. Mohapatra, Multi-agent reinforcement learning: A comprehensive survey, arXiv preprint, 2023, doi: 10.48550/arXiv.2312.10256.
  17. C. Zhu, J. Zhu, W. Si, X. Wang, F. Wang, Multi-agent reinforcement learning with synchronized and decomposed reward automaton synthesized from reactive temporal logic, Knowledge-Based Systems, 2024, 306, 112703, doi: 10.1016/j.knosys.2024.112703.
  18. D. Karimzadeh, Deep reinforcement learning multi-agent systems, International Journal of Modern Achievement in Science, Engineering and Technology, 2024, 2, 56–62, doi: 10.63053/ijset.60.
  19. S. Han, Q. Zhang, W. Jin, Z. Xu, LLM multi-agent systems: Challenges and open problems, arXiv preprint, 2024, doi: 10.48550/arXiv.2402.03578.
  20. Q. Zhao, Y. Zheng, J. Liu, B. Liu, Further analysis for consensus of hybrid multiagent systems: A unified framework, International Journal of Robust and Nonlinear Control, 2022, 31, 8109–8117, doi: 10.1002/rnc.5721.
  21. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, ReAct: Synergizing reasoning and acting in language models, arXiv preprint, 2024, doi: 10.48550/arXiv.2210.03629.
  22. J. Li, H. Shi, K. S. Hwang, A decentralized communication framework based on dual-level recurrence for multi-agent reinforcement learning, arXiv preprint, 2022, doi: 10.48550/arXiv.2202.10612.
  23. J. Hu, Z. Peng, Mathematical methods applied in artificial intelligence and multi-agent systems, MDPI Mathematics, 2024, doi: 10.3390/books978-3-7258-1896-9.
  24. Z. Ma, H. Gong, X. Wang, Multi-agent trajectory decision-making based on transformer models, International Journal of Robust and Nonlinear Control, 2024, doi: 10.1002/rnc.7648.
  25. S. Guo, E. Latif, Y. Zhou, X. Huang, X. Zhai, Using generative AI and multi-agents to provide automatic feedback, arXiv preprint, 2024, doi: 10.48550/arXiv.2411.07407.
  26. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, C. Wang, AutoGen: Enabling next-gen LLM applications via multi-agent conversation framework, Proceedings of the First Conference on Language Modeling (COLM), arXiv preprint, 2024, doi: 10.48550/arXiv.2308.08155.
  27. F. Adobbati, L. Mikulski, Analysing multi-agent systems using 1-safe Petri nets, arXiv preprint, 2023, doi: 10.48550/arXiv.2310.19507.
  28. A. B. Hassouna, H. Chaari, I. Belhaj, LLM-Agent-UMF: LLM-based agent unified modeling framework for seamless design of multi active/passive core-agent architectures, Information Fusion, 2026, 127, Part C, 103865, doi: 10.1016/j.inffus.2025.103865.
  29. K. Osuka, Y. Tsunoda, W. Imahayashi, T. Aotani, Special issue on control and applications of multi-agent systems, Journal of Robotics and Mechatronics, 2024, 36, 507, doi: 10.20965/jrm.2024.p0507.
  30. S. Ren, J. Liu, S. S. Ge, D. Li, HWMP-based secure communication of multi-agent systems, Ad Hoc Networks, 2024, 157, 103456, doi: 10.1016/j.adhoc.2024.103456.
  31. S. G. Ponnambalam, M. N. Janardhanan, G. Rishwaraj, Trust-based decision-making framework for multiagent system, Soft Computing, 2021, 25, 7559–7575, doi: 10.1007/s00500-021-05715-3.
  32. Y. Wang, Privacy in multi-agent systems, arXiv preprint, 2024, doi: 10.48550/arXiv.2403.02631.
  33. T. Song, Y. Tan, Z. Zhu, Y. Feng, Y. C. Lee, Multi-agents are social groups: Investigating social influence of multiple agents in human-agent interactions, arXiv preprint, 2024, doi: 10.48550/arXiv.2411.04578.
  34. M. D. Soorati, E. H. Gerding, E. Marchioni, P. Naumov, T. J. Norman, S. D. Ramchurn, B. Rastegari, From intelligent agents to trustworthy human-centred multiagent systems, AI Communications, 2022, 35, doi: 10.3233/AIC-220127.
  35. S. A. Chernyshev, Problems of multi-agent systems and possible ways to solve them, Vestnik of Russian New University Series, Complex Systems Models Analysis Management, 2023, 231–241, doi: 10.18137/RNU.V9187.23.03.P.231.
  36. H. Xu, J. Yuan, A. Zhou, G. Xu, W. Li, X. Ban, X. Ye, GenAI-powered multi-agent paradigm for smart urban mobility: Opportunities and challenges for integrating large language models (LLMs) and retrieval-augmented generation (RAG) with intelligent transportation systems, arXiv preprint, 2024, doi: 10.48550/arXiv.2409.00494.
  37. A. Rhanizar, Z. El Akkaoui, CM3-VSL: Cooperative multi-goal multi-stage multi-agent VSL traffic control, International Journal of Intelligent Transportation Systems Research, 2024, doi: 10.1007/s13177-024-00426-z.