Skip to main content

Load Balancing

Learn how to use fallback URLs and load balancing with the Proxy.

Fallback URLs​

Configure fallback URLs to try when the primary backend fails:

{
  coreGroups: [
    {
      coreName: 'user-service',
      baseUrl: 'http://localhost:4005',
      routes: [
        {
          name: 'users',
          pathPrefix: '/api/http/user/',
          routeByPath: {
            '': {
              pathPrefix: '/api/http/user/',
              fallbackUrls: [
                'http://localhost:4006/api/http/user/',
                'http://localhost:4007/api/http/user/',
              ],
            },
          },
        },
      ],
    },
  ]
}

Using Cores (Legacy)​

{
  cores: [
    {
      name: 'user-service',
      url: 'http://localhost:4005/api/http/user/',
      pathPrefix: '/users',
      fallbackUrls: [
        'http://localhost:4006/api/http/user/',
        'http://localhost:4007/api/http/user/',
      ],
    },
  ]
}

Fallback Behavior​

The proxy tries URLs in order until one succeeds:

1. Try primary URL: http://localhost:4005
   └─ If success → return response
   └─ If 404 or error → try fallback

2. Try fallback 1: http://localhost:4006
   └─ If success → return response
   └─ If 404 or error → try next fallback

3. Try fallback 2: http://localhost:4007
   └─ If success → return response
   └─ If 404 or error → return error to client

Use Cases​

Migration Support​

When migrating from old service to new:

{
  coreGroups: [
    {
      coreName: 'new-service',
      baseUrl: 'http://localhost:4005',
      routes: [
        {
          name: 'users',
          pathPrefix: '/api/http/user/',
          routeByPath: {
            '': {
              pathPrefix: '/api/http/user/',
              fallbackUrls: ['http://localhost:4006/api/http/user/'],  // old service
            },
          },
        },
      ],
    },
  ]
}

High Availability​

Multiple instances of the same service:

{
  coreGroups: [
    {
      coreName: 'api-primary',
      baseUrl: 'http://api-1:4005',
      routes: [
        {
          name: 'api',
          pathPrefix: '/api/http/',
          routeByPath: {
            '': {
              pathPrefix: '/api/http/',
              fallbackUrls: [
                'http://api-2:4005/api/http/',
                'http://api-3:4005/api/http/',
              ],
            },
          },
        },
      ],
    },
  ]
}

Feature Flags​

Route to different implementations:

const useNewImplementation = process.env.USE_NEW_IMPL === 'true';

{
  coreGroups: [
    {
      coreName: 'feature',
      baseUrl: useNewImplementation
        ? 'http://new-impl:4005'
        : 'http://old-impl:4005',
      routes: [
        {
          name: 'feature',
          pathPrefix: '/api/http/',
          routeByPath: useNewImplementation ? {
            '': {
              pathPrefix: '/api/http/',
              fallbackUrls: ['http://old-impl:4005/api/http/'],  // Fall back to old if new fails
            },
          } : undefined,
        },
      ],
    },
  ]
}

Error Handling​

What Triggers Fallback​

  • HTTP 404 Not Found
  • Connection refused
  • Timeout
  • Network errors

What Does NOT Trigger Fallback​

  • HTTP 400 Bad Request
  • HTTP 401 Unauthorized
  • HTTP 403 Forbidden
  • HTTP 500 Internal Server Error

Only 404 and network errors trigger fallback. Business errors (4xx, 5xx) are returned directly.

Timeout Configuration​

Set timeouts to prevent long waits:

{
  defaultTimeout: 30000,  // 30 seconds global
  coreGroups: [
    {
      coreName: 'fast-service',
      baseUrl: 'http://localhost:4005',
      routes: [
        {
          name: 'fast',
          pathPrefix: '/api/http/',
          timeout: 5000,  // 5 seconds - quick fallback
          routeByPath: {
            '': {
              pathPrefix: '/api/http/',
              fallbackUrls: ['http://localhost:4006/api/http/'],
            },
          },
        },
      ],
    },
  ]
}

Logging​

Enable logging to see fallback behavior:

{
  logRequests: true,
  coreGroups: [...]
}

Output:

[proxy] GET /users/123 → http://localhost:4005/api/http/user/123
[proxy] Primary failed (404), trying fallback...
[proxy] GET /users/123 → http://localhost:4006/api/http/user/123
[proxy] Response: 200 (245ms)

Best Practices​

  1. Order matters: Put the most reliable/preferred backend first

  2. Set appropriate timeouts: Don't wait too long before trying fallback

  3. Monitor fallback usage: High fallback rates indicate problems

  4. Use for graceful degradation: Not as a replacement for proper load balancing

  5. Consider health checks: Combine with health checks for production use

Limitations​

  • Simple round-robin on failure, not true load balancing
  • No health checks (checks on each request)
  • No sticky sessions
  • No weighted routing

For production load balancing, consider using a dedicated load balancer (nginx, HAProxy, AWS ALB) in front of the proxy.

Next Steps​