The AI infrastructure sector is undergoing a transformation as companies shift their focus from large-scale, remote data centers to urban facilities. This trend is driven by the rising importance of inference workloads, which require lower latency and closer proximity to end users. A notable example is Mathpix, a Brooklyn-based AI software company that is deploying Nvidia’s B300 GPU systems in a local colocation facility operated by DataVerge.
The Shift from Training to Inference
Traditionally, AI workloads have centered around training models, which require substantial computational resources. However, with applications moving into production, the focus is changing. Stephen Sopko, an analyst with HyperFrame Research, explains that once a model starts generating revenue, the main challenges shift from raw computational power to factors like round-trip time and data transfer costs. This shift is prompting companies to reassess their infrastructure locations.
Mathpix's deployment at DataVerge illustrates this transition well. Initially, the company relied heavily on cloud services for training but soon recognized the limitations of this model as it aimed to improve inference capabilities. Founder and CEO Nico Jimenez noted a surge in demand for converting PDF documents into machine-readable formats, prompting the search for better infrastructure solutions.
Local Infrastructure Enhances Performance
The combination of local deployments and powerful hardware has proven advantageous for Mathpix. Jimenez described the challenges faced with early iterations of AI hardware in office settings, noting that the noise and cooling requirements of an A100 server made it impractical. This led to a strategic pivot toward a dedicated facility where they could train models and run inference workloads more efficiently.
DataVerge's Brooklyn site is designed to meet the needs of AI workloads, featuring air-cooled GPU deployments capable of handling up to 35 kW per cabinet. As demand for local processing increases, the facility plans to expand its capacity by an additional 3 MW by 2027, further accommodating AI customers' needs.
The Urban Colocation Advantage
Historically, AI infrastructure has been dominated by vast hyperscale campuses in areas like Northern Virginia and Texas, where power and land are abundant. However, the recent shift toward urban colocation facilities reflects changing priorities. Inference workloads require faster data processing and lower latency, making proximity to users essential.
Ray Sidler, CEO of DataVerge, emphasized that many companies are underestimating the role smaller, metro-based data centers can play in the evolving AI sector. As inference operations expand, the demand for these local facilities is likely to rise, providing a counterbalance to the trend of massive, centralized data hubs.
Fragmentation of AI Infrastructure Models
The growing distinction between training and inference workloads is leading to fragmentation in AI infrastructure models. Sopko notes that the market may be evolving into two distinct categories: centralized training facilities that optimize for density and power, and smaller, distributed pods designed for regional users focused on inference.
For Mathpix, this transition has involved a shift toward colocated hardware for critical operational components. As Jimenez indicated, the performance and cost efficiencies of local infrastructure have prompted a reevaluation of their initial reliance on cloud services. Sidler corroborated this shift, revealing that many customers who once turned to Amazon are now exploring hybrid models that blend cloud and local resources.
Looking Ahead
While Mathpix's deployment in Brooklyn may be modest compared to larger, multi-gigawatt AI campuses, it exemplifies a broader trend of AI companies reconstructing their infrastructure around urban environments. As the demands of inference workloads continue to grow, the industry may see a significant pivot toward local facilities that offer enhanced operational control and reduced latency. This evolving landscape suggests that urban colocation centers will play an increasingly vital role in the future of AI infrastructure.
The stories that move AI & crypto markets — before the market reacts.
Free. 7am ET. Five stories. 62,400 readers.
