Multi-Agent Reinforcement Learning with Swarm Coordination for Dynamic Resource Provisioning in Enterprise Analytics Platforms

Abstract

Enterprise analytics platforms expose shared pools of compute, memory, and storage resources to heterogeneous workloads such as interactive queries, streaming pipelines, and batch jobs. These workloads exhibit strong non-stationarity due to diurnal patterns, business events, and evolving user behavior, which makes static or rule-based resource management brittle and often inefficient. At the same time, enterprises impose service level agreements on latency, throughput, and availability that need to be satisfied under cost and energy constraints. Conventional autoscaling controllers rely on local metrics and hand-tuned thresholds, and they often treat the platform as a monolithic system, ignoring the spatial structure of clusters and the coupling between tenants. Multi-agent reinforcement learning provides a way to learn adaptive policies from interaction, while swarm coordination offers a decentralized mechanism for aligning behavior across large agent populations.   This paper studies dynamic resource provisioning in enterprise analytics platforms through a swarm-coordinated multi-agent reinforcement learning formulation. The platform is modeled as a set of interacting resource pools, each controlled by an agent that observes local performance signals and takes scaling actions. Swarm coordination mechanisms propagate aggregate load and congestion information across agents using lightweight neighborhood communication, which reduces oscillations and improves global constraint satisfaction without centralizing the control logic. A linear model of resource consumption and performance is integrated into the learning process to structure value functions and constrain exploration. The study investigates stability, scalability, and empirical behavior of the proposed framework across a range of workload patterns, highlighting trade-offs between responsiveness, resource utilization, and service level adherence.