Rate Limiter in Triton: Managing Resource Allocation for MLOps

1. Introduction

Deploying machine learning models can be tricky. One big challenge is managing computer resources like memory and processing power (GPU utilization). When you're running multiple models at once::

• Your GPU VRam might run out of memory
• Some models might use up all the resources, leaving others waiting
• The whole system could slow down or crash This is where Triton Inference Server comes in handy. Triton is a tool that helps deploy and run machine learning models. It has a feature called the Rate Limiter, which helps manage resources better. In this tutorial, we'll learn about Triton and its Rate Limiter. We'll start with the basics and then see how the Rate Limiter can solve resource problems.

2. Triton Inference Server Fundamentals

What is Triton Inference Server?

Triton Inference Server is like a helpful manager for your machine learning models. It's a software that:

• Runs your models
• Handles requests to use these models
• Manages resources for smooth operation

Key Features of Triton

• Supports many types of models (TensorFlow, PyTorch, ONNX, etc.)
• Can run multiple models at the same time
• Allows models to run on CPUs or GPUs
• Has tools to manage resources (like the Rate Limiter we'll learn about)

Basic Architecture

Let's imagine Triton as a building with many rooms:

• Each room can hold a different model
• There's a front desk (API) where requests come in
• The building manager (Triton) decides which room (model) to use and when

3. Getting Started with Triton

Let's set up a simple Triton server and run a basic model.

Installation

1. The easiest way to get started is using Docker. Install Docker if you haven't already.
2. Pull the Triton Docker image:

1docker pull nvcr.io/nvidia/tritonserver:23.09-py3-sdk

Running a Simple Triton Server

1. Create a folder for your models:

   1mkdir -p models/simple_model/1

2. For this example, let's use a pre-trained ONNX model. Download a simple model (like a small image classifier) and place it in models/simple_model/1/model.onnx.
3. Create a configuration file models/simple_model/config.pbtxt:

   1name: "simple_model"
   2platform: "onnxruntime_onnx"
   3max_batch_size: 8
   4input [
   5  {
   6    name: "input"
   7    data_type: TYPE_FP32
   8    dims: [ 3, 224, 224 ]
   9  }
   10]
   11output [
   12  {
   13    name: "output"
   14    data_type: TYPE_FP32
   15    dims: [ 1000 ]
   16  }
   17]

4. Start the Triton server:

1docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v $(pwd)/models:/models nvcr.io/nvidia/tritonserver:21.09-py3 tritonserver --model-repository=/models

You now have a Triton server running with a simple model!

Note:

• The --gpus all flag ensures Triton can access all available GPUs.
• Use nvidia-smi to verify GPU availability before starting Triton.

4. Understanding the Rate Limiter

What is a Rate Limiter?

A Rate Limiter is like a traffic controller for your models. It decides:

• How many requests each model can handle at once
• Which models get to run when resources are limited

How Triton's Rate Limiter Works

Triton's Rate Limiter uses two main concepts:

1. Resources: These are things your models need, like GPU memory or CPU power.
2. Priorities: This decides which models are more important when resources are limited. The Rate Limiter keeps track of available resources and assigns them to models based on their needs and priorities.

5. Configuring the Rate Limiter in Triton

Let's set up the Rate Limiter for our Triton server.

Defining Resources

Resources are identified by a unique name and a count indicating the number of copies of the resource. Here's how to define resources in your model's config file (config.pbtxt):

1instance_group [
2  {
3    count: 1
4    kind: KIND_GPU
5    gpus: [ 0, 1, 2 ]
6    rate_limiter {
7      resources [
8        {
9          name: "R1"
10          count: 4
11        },
12        {
13          name: "R2"
14          global: True
15          count: 2
16        }
17      ]
18      priority: 2
19    }
20  }
21]

In this example:

• The model instance requires 4 units of resource "R1" and 2 units of resource "R2".
• "R2" is specified as a global resource, meaning it's shared across all devices.
• This configuration creates 3 model instances, one on each GPU (0, 1, and 2). The three instances won't contend for "R1" among themselves as it's local to each device. However, they will contend for "R2" because it's a global resource shared across the system. Note: When configuring the Rate Limiter, it's important to understand what the resource numbers means that from the pool of all resources (global or per-device), this model requires 4 R1 resources to execute. For example, if you have 10 R1 resources for a device in total, and you have 3 instances of model A, each requiring 4 R1 resources, only two instances of model A can be executed simultaneously.

Setting Total Resources

By default, the available number of resource copies is the maximum across all model instances that list that resource. For example, if you have three model instances with the following resource requirements:

A: [R1: 4, R2: 4]
B: [R2: 5, R3: 10, R4: 5]
C: [R1: 1, R3: 7, R4: 2]

Triton will create the following resources:

R1: 4
R2: 5
R3: 10
R4: 5

You can override these defaults using the --rate-limit-resource option when starting Triton:

1docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v $(pwd)/models:/models nvcr.io/nvidia/tritonserver:21.09-py3 tritonserver --model-repository=/models --rate-limit-resource=R1:10 --rate-limit-resource=R2:5:0 --rate-limit-resource=R2:8:1 --rate-limit-resource=R3:2:global

This command sets up the following resources:

GLOBAL   => [R3: 2]
DEVICE 0 => [R1: 10, R2: 5]
DEVICE 1 => [R1: 10, R2: 8]

--rate-limit-resource Flags:

• --rate-limit-resource=R1:10:
  • R1: The name of the resource.
  • 10: The limit for this specific resource.
• --rate-limit-resource=R2:5:0:
  • R2: The resource name.
  • 5: The limit for this specific resource.
  • 0: The resource's instance index (e.g., the 0th instance of R2).
• --rate-limit-resource=R2:8:1:
  • Similar to the above, this specifies that the 1st instance of R2 has a limit of 8.
• --rate-limit-resource=R3:2:global:
  • R3: Another resource.
  • 2: Limit for this resource.
  • global: Instead of being instance-specific, this limit applies globally across all instances of this resource.

Configuring Priorities:

Priority serves as a weighting value for scheduling across all model instances. An instance with priority 2 will be given half the number of scheduling chances as an instance with priority 1. You can set the priority in the rate limiter configuration:

rate_limiter {
  priority: 2
}

Example:

Imagine you are running two model instances on the same Triton server:

1. Instance A (latency-sensitive): Should handle requests faster, so you set priority: 1.
2. Instance B (less critical): Doesn't need as much scheduling attention, so you set priority: 2. Here’s how you can configure this in Triton’s config.pbtxt file:

1instance_group [
2      {
3        kind: KIND_GPU
4    count: 1
5    rate_limiter {
6          priority: 1
7    }
8  },
9  {
10        kind: KIND_CPU
11    count: 1
12    rate_limiter {
13          priority: 2
14    }
15  }
16]

Instance A (GPU) with priority: 1 will be scheduled twice as often as Instance B (CPU) with priority: 2, giving it more chances to process requests faster.

7. Practical Example: Configuring Rate Limiter for Multiple Models

Here is a practical example of configuring the Rate Limiter for multiple models.

Step 1: Calculate Total GPU Memory

First, determine the total memory of your GPU. Let's say we have a GPU with 16GB (16,384 MB) of memory.

Step 2: Calculate Base Memory Usage

Calculate the memory usage of each model when loaded into Triton (not serving any inference requests). For this example, let's assume:

• Model A: 500 MB
• Model B: 700 MB
• Model C: 600 MB Total base memory usage: 500 + 700 + 600 = 1,800 MB

Step 3: Calculate Available Memory

Subtract the base memory usage from the total GPU memory: 16,384 MB - 1,800 MB = 14,584 MB This 14,584 MB is our available memory for inference (let's call this value m).

Step 4: Determine Maximum Memory Usage

Find out the maximum memory usage of each model instance during inference. For this example, let's assume:

• Model A: 1,000 MB
• Model B: 1,500 MB
• Model C: 1,200 MB

Step 5: Set Resource Values

Now, we'll set the R1 values for each model based on their maximum memory usage. We'll use 1 R1 unit to represent 100 MB of GPU memory:

instance_group [
  {
    count: 1
    kind: KIND_GPU
    rate_limiter {
      resources [
        {
          name: "R1"
          count: 10  # Model A: 1,000 MB / 100 MB per unit = 10 units
        }
      ]
    }
  }
]

Similarly, for Model B, count would be 15, and for Model C, it would be 12.

Step 6: Set Total Per-Device Resource Value

Set the total per-device R1 value to m (our available memory for inference):

1tritonserver --rate-limit-resource=R1:145 --model-repository=/path/to/model_repository

Here, 145 represents 14,584 MB / 100 MB per unit, rounded down.

Complete Configuration Example

Here's how the complete configuration might look:

# Model A config.pbtxt
name: "model_a"
platform: "onnxruntime_onnx"
max_batch_size: 8
instance_group [
      {
        count: 1
    kind: KIND_GPU
    rate_limiter {
          resources [
            {
              name: "R1"
          count: 10
        }
      ]
    }
  }
]
# Model B config.pbtxt
name: "model_b"
platform: "onnxruntime_onnx"
max_batch_size: 8
instance_group [
      {
        count: 1
    kind: KIND_GPU
    rate_limiter {
          resources [
            {
              name: "R1"
          count: 15
        }
      ]
    }
  }
]
# Model C config.pbtxt
name: "model_c"
platform: "onnxruntime_onnx"
max_batch_size: 8
instance_group [
      {
        count: 1
    kind: KIND_GPU
    rate_limiter {
          resources [
            {
              name: "R1"
          count: 12
        }
      ]
    }
  }
]

With this configuration:

• Model A can use up to 1,000 MB of GPU memory during inference
• Model B can use up to 1,500 MB
• Model C can use up to 1,200 MB Note: The total usage across all models won't exceed 14,500 MB, leaving some buffer and accounting for the base memory usage

8. Best Practices and Tips

• Use the Model Analyzer (https://github.com/triton-inference-server/model_analyzer) to help with memory profiling steps.
• Monitor your system's performance and adjust resource allocations as needed.
• Remember to account for base memory usage of loaded models.
• Leave some buffer in your calculations to account for system overhead and variations in memory usage.
• For models with highly variable memory usage, consider setting the resource count to the maximum expected usage to ensure stability.

9. Advanced Rate Limiter Configurations

Global vs. Per-Device Resources

• Per-device resources: By default, resources are per-device. Resource requirements for a model instance are enforced against the resources associated with the device where the model instance runs.
• Global resources: Instead of creating resource copies per-device, a global resource has a single copy shared across the entire system. You can specify a global resource in the model configuration or when starting Triton.

Fine-tuning Priorities

In a resource-constrained system, there will be contention for resources among model instances. The priority setting helps determine which model instance to select for the next execution. Lower priority values indicate higher priority (more scheduling chances).

10. Conclusion

The Rate Limiter is a powerful tool in Triton that helps you:

• Manage resources efficiently
• Run multiple models safely
• Improve system stability