HTTP is a request-response protocol. The client asks, the server answers, the connection closes. For most web applications, this model works perfectly. But when you need the server to push data to the client—live notifications, real-time collaboration, streaming inference results, chat messages, live dashboards—you need a persistent, bidirectional connection. WebSockets provide this. At Harbor Software, we built a WebSocket layer for streaming ML inference results to the browser: the user submits a query, and tokens stream back as the model generates them, word by word, creating a responsive experience that feels like the AI is thinking in real time rather than computing behind a spinner. This post covers the architecture patterns, connection management, scaling strategies, and the production-specific challenges we encountered building this system.

When You Actually Need WebSockets

WebSockets add meaningful complexity to your application: persistent connections require different scaling strategies than stateless HTTP, you need heartbeat mechanisms to detect dead connections, reconnection logic on the client, and a pub/sub system if you scale beyond a single server. Before reaching for them, verify that simpler alternatives do not solve your problem:

• Short polling: The client makes an HTTP request every N seconds to check for new data. If your data updates every 5–30 seconds and slight delay is acceptable, polling is dramatically simpler and more resilient than WebSockets. A 5-second polling interval with HTTP/2 connection reuse adds minimal overhead. We use polling for our admin dashboard metrics because 5-second freshness is perfectly adequate for graphs and counters.
• Long polling: The client makes an HTTP request that the server holds open until new data is available (or a timeout occurs). This gives near-real-time updates without the complexity of WebSockets. The trade-off is that each update requires a new HTTP request, which adds latency compared to an already-open WebSocket.
• Server-Sent Events (SSE): A one-directional streaming protocol from server to client that runs over standard HTTP. SSE handles reconnection automatically (the browser re-establishes the connection with a Last-Event-ID header), works through HTTP proxies without special configuration, and is supported by all modern browsers. SSE is the right choice for notification feeds, live dashboards, and progress indicators—any case where data flows only from server to client.
• WebSockets: Bidirectional real-time communication where both client and server send messages independently. Use WebSockets when the client needs to send control messages during an active stream (pause, cancel, parameter update), when you need binary data transfer, or when you need sub-50ms latency for interactive applications.

We chose WebSockets for inference streaming because our application requires bidirectional communication: the server streams tokens to the client, and the client can send control messages mid-stream—”cancel this generation,” “regenerate with higher temperature,” “switch to a different model.” SSE would handle the server-to-client token stream, but the client-to-server control messages would require a separate HTTP endpoint, creating a coordination problem between two communication channels that WebSockets handle natively in one connection.

The Protocol Layer

Raw WebSocket messages are unstructured strings or binary blobs. Without a protocol layer, you end up with ad-hoc string parsing, no type safety, and no way to distinguish between different message types. We define a typed message protocol with a type discriminator field:

// shared/ws-types.ts (used by both client and server)
interface BaseMessage {
  type: string;
  id: string;       // Unique message ID for acknowledgment and correlation
  timestamp: number; // Unix timestamp in milliseconds
}

// Client-to-server messages
interface StartInferenceMessage extends BaseMessage {
  type: 'inference.start';
  payload: {
    prompt: string;
    modelId: string;
    maxTokens: number;
    temperature: number;
    streamTokens: boolean;
  };
}

interface CancelInferenceMessage extends BaseMessage {
  type: 'inference.cancel';
  payload: {
    inferenceId: string;
  };
}

interface PingMessage extends BaseMessage {
  type: 'ping';
}

// Server-to-client messages
interface TokenMessage extends BaseMessage {
  type: 'inference.token';
  payload: {
    inferenceId: string;
    token: string;
    index: number;
    finishReason: null; // Not finished yet
  };
}

interface InferenceCompleteMessage extends BaseMessage {
  type: 'inference.complete';
  payload: {
    inferenceId: string;
    totalTokens: number;
    latencyMs: number;
    finishReason: 'stop' | 'max_tokens' | 'cancelled';
  };
}

interface ErrorMessage extends BaseMessage {
  type: 'error';
  payload: {
    code: string;
    message: string;
    inferenceId?: string;
  };
}

interface PongMessage extends BaseMessage {
  type: 'pong';
}

type ClientMessage = StartInferenceMessage | CancelInferenceMessage | PingMessage;
type ServerMessage = TokenMessage | InferenceCompleteMessage | ErrorMessage | PongMessage;

This protocol gives us several things that raw WebSocket messages lack: the type field enables discriminated union pattern matching on both client and server; the id field enables request-response correlation (the client can match a inference.complete to its original inference.start); and the timestamp field enables end-to-end latency measurement. Sharing these types between client and server (we publish them as a shared npm package) ensures both sides agree on the protocol and TypeScript catches protocol mismatches at compile time.

Server Implementation

Our server uses the ws library for Node.js, which provides a low-level WebSocket implementation with excellent performance. Here is the core structure:

import { WebSocketServer, WebSocket } from 'ws';
import { v4 as uuid } from 'uuid';
import { IncomingMessage } from 'http';

interface Client {
  id: string;
  ws: WebSocket;
  userId: string;
  activeInferences: Map<string, AbortController>;
  lastPong: number;
}

class InferenceStreamServer {
  private wss: WebSocketServer;
  private clients: Map<string, Client> = new Map();
  private heartbeatInterval: NodeJS.Timer;

  constructor(port: number) {
    this.wss = new WebSocketServer({ port });
    this.wss.on('connection', (ws, req) => this.handleConnection(ws, req));
    
    // Heartbeat every 30 seconds to detect dead connections
    this.heartbeatInterval = setInterval(() => this.heartbeat(), 30000);
    
    console.log(`WebSocket server listening on port ${port}`);
  }

  private handleConnection(ws: WebSocket, req: IncomingMessage) {
    // Authenticate from query parameter
    const url = new URL(req.url || '', 'http://localhost');
    const token = url.searchParams.get('token');
    const userId = this.verifyToken(token);
    
    if (!userId) {
      ws.close(4001, 'Authentication failed');
      return;
    }

    const client: Client = {
      id: uuid(),
      ws,
      userId,
      activeInferences: new Map(),
      lastPong: Date.now()
    };
    
    this.clients.set(client.id, client);
    console.log(`Client connected: ${client.id} (user: ${userId}, total: ${this.clients.size})`);

    ws.on('message', (data) => this.handleMessage(client, data));
    ws.on('close', (code, reason) => this.handleDisconnect(client, code, reason.toString()));
    ws.on('error', (err) => {
      console.error(`Client ${client.id} error:`, err.message);
      // Error is always followed by close event, cleanup happens there
    });
    ws.on('pong', () => { client.lastPong = Date.now(); });
  }

  private async handleMessage(client: Client, data: any) {
    let message: ClientMessage;
    try {
      message = JSON.parse(data.toString());
    } catch {
      this.sendError(client, 'PARSE_ERROR', 'Invalid JSON');
      return;
    }
    
    // Validate message has required fields
    if (!message.type || !message.id) {
      this.sendError(client, 'INVALID_MESSAGE', 'Missing type or id field');
      return;
    }
    
    switch (message.type) {
      case 'inference.start':
        await this.startInference(client, message);
        break;
      case 'inference.cancel':
        this.cancelInference(client, message);
        break;
      case 'ping':
        this.send(client, { type: 'pong', id: message.id, timestamp: Date.now() });
        break;
      default:
        this.sendError(client, 'UNKNOWN_TYPE', `Unknown message type: ${(message as any).type}`);
    }
  }

The activeInferences map tracks running inference jobs with their AbortController, which allows clean cancellation when the client sends a cancel message or disconnects. This is critical for resource management—without it, a disconnected client’s inference job continues consuming GPU resources until it completes naturally.

  private async startInference(client: Client, message: StartInferenceMessage) {
    // Rate limiting: max 3 concurrent inferences per client
    if (client.activeInferences.size >= 3) {
      this.sendError(client, 'RATE_LIMIT', 'Maximum 3 concurrent inferences', undefined);
      return;
    }
    
    const inferenceId = uuid();
    const abortController = new AbortController();
    client.activeInferences.set(inferenceId, abortController);
    
    const startTime = Date.now();
    let tokenIndex = 0;
    
    try {
      const stream = await this.inferenceEngine.stream({
        prompt: message.payload.prompt,
        modelId: message.payload.modelId,
        maxTokens: message.payload.maxTokens,
        temperature: message.payload.temperature,
        signal: abortController.signal
      });
      
      for await (const token of stream) {
        // Check if cancelled (AbortController handles this, but belt-and-suspenders)
        if (abortController.signal.aborted) break;
        
        // Check if client still connected
        if (client.ws.readyState !== WebSocket.OPEN) break;
        
        this.send(client, {
          type: 'inference.token',
          id: uuid(),
          timestamp: Date.now(),
          payload: { inferenceId, token, index: tokenIndex++, finishReason: null }
        });
      }
      
      const finishReason = abortController.signal.aborted ? 'cancelled' : 'stop';
      
      this.send(client, {
        type: 'inference.complete',
        id: uuid(),
        timestamp: Date.now(),
        payload: {
          inferenceId,
          totalTokens: tokenIndex,
          latencyMs: Date.now() - startTime,
          finishReason
        }
      });
    } catch (err) {
      if (!abortController.signal.aborted) {
        this.sendError(client, 'INFERENCE_FAILED', (err as Error).message, inferenceId);
      }
    } finally {
      client.activeInferences.delete(inferenceId);
    }
  }

  private cancelInference(client: Client, message: CancelInferenceMessage) {
    const controller = client.activeInferences.get(message.payload.inferenceId);
    if (controller) {
      controller.abort();
      // The for-await loop in startInference will detect the abort and send completion
    }
  }

  private handleDisconnect(client: Client, code: number, reason: string) {
    // Cancel all active inferences for this client
    for (const [id, controller] of client.activeInferences) {
      controller.abort();
    }
    client.activeInferences.clear();
    this.clients.delete(client.id);
    console.log(`Client disconnected: ${client.id} (code: ${code}, total: ${this.clients.size})`);
  }

Connection Lifecycle Management

WebSocket connections are long-lived, which introduces failure modes that HTTP’s stateless model avoids entirely:

Dead connections. When a client’s network drops abruptly (laptop lid closes, WiFi disconnects, mobile network switches), the TCP connection may not close cleanly. The server keeps the connection in its active set, continues attempting to send messages (which silently buffer), and the resources associated with that client (memory, inference slots) remain allocated. Our heartbeat mechanism (ping every 30 seconds, terminate if no pong within 60 seconds) detects and cleans up dead connections:

  private heartbeat() {
    const now = Date.now();
    for (const [id, client] of this.clients) {
      // If no pong received in 60 seconds, connection is dead
      if (now - client.lastPong > 60000) {
        console.log(`Client ${id} failed heartbeat (last pong: ${now - client.lastPong}ms ago)`);
        client.ws.terminate(); // Hard close, triggers 'close' event
        continue;
      }
      // Send ping to check if connection is alive
      if (client.ws.readyState === WebSocket.OPEN) {
        client.ws.ping();
      }
    }
  }

Without heartbeats, dead connections accumulate silently. In our first week of production, before we added heartbeats, we observed 47 “zombie” connections—clients that had disconnected hours ago but were still in the server’s active set, each holding allocated inference capacity. The heartbeat mechanism cleaned this up completely.

Client reconnection with exponential backoff. The client must handle disconnection and reconnection transparently. When a WebSocket closes unexpectedly, the client should attempt to reconnect with increasing delays to avoid thundering herd problems:

class ReconnectingWebSocket {
  private ws: WebSocket | null = null;
  private reconnectAttempts = 0;
  private maxReconnectAttempts = 10;
  private baseDelay = 1000;
  private handlers: Map<string, Function[]> = new Map();
  private pendingMessages: string[] = [];

  connect(url: string) {
    this.ws = new WebSocket(url);
    
    this.ws.onopen = () => {
      console.log('WebSocket connected');
      this.reconnectAttempts = 0;
      
      // Flush any messages that were queued during reconnection
      for (const msg of this.pendingMessages) {
        this.ws!.send(msg);
      }
      this.pendingMessages = [];
      
      this.emit('connected');
    };
    
    this.ws.onmessage = (event) => {
      try {
        const message = JSON.parse(event.data);
        this.emit(message.type, message);
      } catch (e) {
        console.error('Failed to parse WebSocket message:', e);
      }
    };
    
    this.ws.onclose = (event) => {
      if (event.code === 4001) {
        // Auth failure - don't reconnect, redirect to login
        this.emit('auth_failure');
        return;
      }
      if (event.code === 1000) {
        // Normal closure - don't reconnect
        return;
      }
      this.scheduleReconnect(url);
    };
  }

  private scheduleReconnect(url: string) {
    if (this.reconnectAttempts >= this.maxReconnectAttempts) {
      this.emit('max_retries_exceeded');
      return;
    }
    
    // Exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s (capped)
    const delay = Math.min(
      this.baseDelay * Math.pow(2, this.reconnectAttempts),
      32000
    );
    // Add jitter: +/- 25% to prevent thundering herd
    const jitter = delay * (0.75 + Math.random() * 0.5);
    
    this.reconnectAttempts++;
    console.log(`Reconnecting in ${Math.round(jitter)}ms (attempt ${this.reconnectAttempts})`);
    
    setTimeout(() => this.connect(url), jitter);
  }
  
  send(message: object) {
    const data = JSON.stringify(message);
    if (this.ws?.readyState === WebSocket.OPEN) {
      this.ws.send(data);
    } else {
      // Queue message for sending after reconnection
      this.pendingMessages.push(data);
    }
  }
}

The jitter (randomizing the reconnection delay by +/- 25%) is essential for production systems. If your server restarts and 500 clients reconnect simultaneously, without jitter they all hit the server at the same exponential backoff intervals (1s, 2s, 4s…), creating periodic load spikes that can cascade into further failures. Jitter spreads the reconnections across the delay window, producing smooth, manageable load.

The message queuing (pendingMessages) prevents messages from being lost during brief disconnections. If the user clicks “cancel” while the WebSocket is reconnecting, the cancel message is queued and sent as soon as the connection is re-established.

Scaling Beyond One Server

A single WebSocket server handles around 10,000–50,000 concurrent connections (depending on message rate, message size, and server resources). When you need to scale beyond this, you hit the fundamental challenge: WebSocket connections are stateful and pinned to a specific server process. HTTP’s statelessness means any server can handle any request; WebSocket’s statefulness means each client is bound to one server for the duration of the connection.

If client A is connected to server 1 and you need to send a notification to client A from a background job running on server 2, you need an inter-server communication mechanism. The standard solution is a Redis pub/sub backbone:

import Redis from 'ioredis';

class ScaledWebSocketServer {
  private pub: Redis;
  private sub: Redis;
  private localClients: Map<string, Client> = new Map();
  private serverId: string;

  constructor() {
    this.serverId = uuid();
    this.pub = new Redis(process.env.REDIS_URL);
    this.sub = new Redis(process.env.REDIS_URL);
    
    // Subscribe to channels for messages targeting this server's clients
    this.sub.subscribe('ws:broadcast');
    this.sub.on('message', (channel, message) => {
      const parsed = JSON.parse(message);
      // Don't echo messages we published ourselves
      if (parsed._sourceServer === this.serverId) return;
      this.deliverToLocalClients(parsed);
    });
  }

  // Send a message to a specific user (who might be on any server)
  async sendToUser(userId: string, message: ServerMessage) {
    // Try local delivery first (fastest path)
    const localClient = this.findLocalClientByUserId(userId);
    if (localClient) {
      this.send(localClient, message);
      return;
    }
    // User is not on this server - publish to Redis for other servers
    this.pub.publish('ws:broadcast', JSON.stringify({
      ...message,
      _targetUserId: userId,
      _sourceServer: this.serverId
    }));
  }

  private deliverToLocalClients(parsed: any) {
    const targetUserId = parsed._targetUserId;
    if (targetUserId) {
      // Targeted message: deliver only to the specified user
      const client = this.findLocalClientByUserId(targetUserId);
      if (client) {
        const { _targetUserId, _sourceServer, ...message } = parsed;
        this.send(client, message);
      }
    } else {
      // Broadcast: deliver to all local clients
      for (const client of this.localClients.values()) {
        this.send(client, parsed);
      }
    }
  }
}

Redis pub/sub adds 1–2ms of latency per message but enables horizontal scaling to hundreds of thousands of concurrent connections across multiple servers behind a load balancer. We are not yet at this scale (peak ~200 concurrent connections), but we built the pub/sub architecture from the start because retrofitting it onto a working WebSocket server is significantly harder than building it in. The code path for single-server deployment is the same—the Redis pub/sub just has one subscriber—so there is no operational cost to having the infrastructure in place early.

Authentication and Security

WebSocket connections bypass most HTTP middleware because the protocol upgrade from HTTP to WebSocket happens before most middleware runs. CORS headers, CSRF tokens, and cookie-based authentication work differently (or not at all) for WebSocket connections. Authentication must be handled explicitly during the connection handshake.

We use short-lived tokens: the client obtains a WebSocket-specific JWT from an authenticated HTTP endpoint (POST /api/ws-token), then connects using that token in the query string. The token has a 5-minute expiry, limiting the exposure window if it leaks. The server validates the token during the connection handshake and rejects invalid or expired tokens with close code 4001 (custom application error). This approach keeps secrets out of WebSocket message payloads and works through HTTP proxies and load balancers that might inspect query parameters but cannot modify WebSocket frames.

Monitoring and Observability

WebSocket connections are harder to observe than HTTP requests because they are long-lived and bidirectional. A traditional HTTP access log shows one line per request with status code and latency. A WebSocket connection might last hours and exchange thousands of messages. We track these metrics, exported to Prometheus and visualized in Grafana:

• Active connections (gauge): Current connected clients, segmented by user tier
• Connection duration (histogram): Distribution of connection lifetimes—helps detect clients that connect and immediately disconnect (usually authentication failures)
• Messages per second (counter): Inbound and outbound, segmented by message type
• Token streaming latency (histogram): Time between generating a token and the client acknowledging receipt
• Error rate (counter): Segmented by error code (parse errors, auth failures, inference failures, rate limit hits)
• Send buffer backlog (histogram): Bytes queued for sending per connection—detects slow clients falling behind

The “active connections” gauge is most important for capacity planning. The “send buffer backlog” histogram is most important for detecting performance problems—a client with a growing backlog is receiving data slower than the server generates it, which eventually leads to either memory pressure or dropped messages.

Conclusion

WebSocket applications are architecturally different from HTTP applications in ways that affect every layer of the stack. Connections are stateful (requiring explicit lifecycle management), long-lived (requiring heartbeats and reconnection), and bidirectional (requiring a message protocol with type discrimination and correlation). Before choosing WebSockets, verify that SSE or polling cannot solve your problem—they are simpler, more resilient, and sufficient for the majority of “real-time” use cases. If WebSockets are the right tool, invest upfront in a typed message protocol, implement heartbeats from day one, design for horizontal scaling with pub/sub even if you do not need it yet, and build monitoring that tracks connection lifecycle metrics alongside traditional request metrics. The patterns in this post have served us well for streaming ML inference results, and they generalize to chat, collaboration, live dashboards, and any application where server-initiated, low-latency communication is a core requirement.