在SageMaker AI上对小型LLM推断进行基准测试：G7 vs G5和G6

Choosing the right GPU instance for large language model (LLM) inference is one of the most impactful decisions you make when deploying generative AI at scale. A single generation jump can slash latency, increase throughput, and reduce cost-per-token. However, the real-world magnitude of those gains depends on model architecture, quantization format, and workload shape.

In this post, we benchmark two representative 30B Mixture-of-Experts (MoE) models across three GPU instance families on Amazon SageMaker AI Inference using Amazon SageMaker AI Generative AI inference recommendation. This feature provides both recommendations for throughput, cost, and latency, and benchmarking for common metrics such as time to first token and latency.

Using this feature, we demonstrate how the new G7 instances powered by NVIDIA Blackwell GPUs deliver measurable gains in throughput, latency, and cost-per-token.

Use case 1: AI coding assistant

Deploy Qwen3-Coder-30B for enterprise coding tasks such as code generation, debugging, refactoring, and developer copilots. Benchmark ml.g5.12xlarge (A10G), ml.g6.12xlarge (L4), and ml.g7.12xlarge (RTX PRO 4500 Blackwell) instances using the Amazon SageMaker AI DJL Large Model Inference (LMI) container to compare price and performance.

The G5 and G6 configurations each use four GPUs with 96 GB of aggregate GPU memory, while G7 uses two GPUs with 64 GB. This comparison shows how G7 performs with half the number of accelerators and less total GPU memory.

Use case 2: Enterprise AI assistant

Deploy NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4 for reasoning, question answering, summarization, and agentic workloads. Use Amazon SageMaker AI Generative AI Inference Recommendations with vLLM to evaluate G6 (L4), G6e (L40S), and G7 (RTX PRO 4500 Blackwell) configurations and identify the best price-performance option.

For the 12xlarge configurations, G6 provides four GPUs with 96 GB of aggregate GPU memory, G6e provides four GPUs with 192 GB, and G7 provides two GPUs with 64 GB. Generative AI Inference Recommendations automates the benchmarking and configuration selection across these GPU options.

Together, these use cases demonstrate two complementary ways to evaluate G7: direct benchmarking with LMI and automated optimization with Generative AI Inference Recommendations and vLLM. By varying the models, serving stacks, GPU configurations, and workloads, the benchmarks show when G7 can provide better price-performance than prior-generation GPU instances.

Solution overview

Before we walk through the use cases, we explain how Amazon SageMaker AI generative AI inference recommendations simplifies the process of finding the right production configuration. You provide your model, expected workload, and optimization goal. Amazon SageMaker AI then evaluates candidate configurations on real GPU infrastructure and returns validated, deployment-ready recommendations based on actual performance.

You can connect each workflow directly to the two use cases:

1. Benchmarking workflow: When you already have an endpoint

Use benchmarking when you have a model deployed to an Amazon SageMaker AI endpoint and want to understand how it performs under your expected workload. In our Qwen3-Coder-30B use case, we will first show you how to deploy the model to Amazon SageMaker AI endpoints on different GPU instances and then benchmark those endpoints to compare latency, throughput, and price-performance as shown in the diagram.

2. Recommendation workflow: When you want Amazon SageMaker AI inference recommendation to recommend the configuration

Use the recommendation workflow when you want help determining which deployment configuration is best for your model and workload. In our NVIDIA Nemotron-3-Nano-30B use case, we will define the model, candidate instance families, workload, and optimization objective. Amazon SageMaker AI then evaluates candidate configurations, benchmarks for viable options, and returns ranked recommendations based on metrics such as cost, latency, and throughput. You can review the recommendations and deploy the configuration that best meets your production requirements as shown in the following diagram:

Instance hardware comparison

Both cases use instances from the AWS EC2 G5, G6, G6e, and G7 families. Each family pairs a different NVIDIA GPU generation with varying amounts of memory, bandwidth, and networking. The key differences that affect Mixture-of-Experts (MoE) inference performance are:

• GPU memory capacity (which determines whether a model fits on a single GPU).
• Memory bandwidth (which governs decode speed).
• Native support for low-precision formats like FP8 and NVFP4.

Why memory bandwidth matters for MoE models: MoE architectures are memory-bandwidth bound during token generation (decoding) because each token activates only a small subset of experts. Higher memory bandwidth directly reduces inter-token latency and increases throughput.

Why NVFP4 matters: NVFP4 is a 4-bit floating-point format introduced by NVIDIA with the Blackwell GPU architecture. NVIDIA’s native FP4 quantization on Blackwell Tensor Cores reduces model size to approximately 4 bits per weight with minimal quality loss.

Note: Only G7 instances have native FP4 Tensor Core support. Other generations such as G5 and G6 run NVFP4 weights without hardware acceleration, which puts G7 at a structural advantage for MoE deployment choice.

Prerequisites

• An AWS account with Amazon SageMaker AI access.
• An AWS Identity and Access Management (IAM) execution role with Amazon SageMaker AI and Amazon Simple Storage Service (Amazon S3) permissions.
• Instance quota for your target instance types in your AWS Region.
• Python 3.10+ with the Amazon SageMaker Python SDK (sagemaker>=3.16.0)
• Model artifacts uploaded to Amazon S3

Note: G7 is generally available in US East (Ohio) and US West (Oregon) only.

Use case 1: Benchmark Qwen3-Coder-30B across GPU instances

In this use case, we benchmark Qwen3-Coder-30B-A3B-Instruct-FP8 across three GPU instance types: ml.g5.12xlarge, ml.g6.12xlarge, and ml.g7.12xlarge to determine which configuration provides the best performance for our coding workload using this walkthrough notebook.

We use the same model, DJL Large Model Inference (LMI) 28.0 serving container, and workload across all three configurations to provide an apples-to-apples comparison. The accompanying notebook contains the complete code for model configuration, endpoint deployment, and benchmarking.

The workflow consists of three steps:

• Configure the benchmark
• Run the benchmark
• Compare the results

Step 1: Configure the benchmark

We first define the model and workload that we want to evaluate. For this test, we use Qwen3-Coder-30B-A3B-Instruct-FP8, a Mixture-of-Experts (MoE) coding model, and deploy it using the DJL Large Model Inference (LMI) 28.0 container.

We evaluate the same model across the following Amazon SageMaker AI instance types:

We use the same serving and workload configuration for each endpoint so that the underlying instance type is the primary variable in the comparison.

The accompanying notebook provides the complete configuration and deployment code for each endpoint.

Step 2: Run the benchmark

After the endpoints are available, we define a synthetic workload that represents our coding inference scenario and run the Amazon SageMaker AI benchmark against each endpoint.

For this example, the workload contains 100 requests at a concurrency of four, with an average of 128 input tokens and 128 output tokens per request.

At a high level, the benchmark is started using the following Amazon SageMaker Python SDK operations:

from sagemaker.serve import start_benchmark, Workload

workload = Workload.synthetic(
    tokenizer=HF_MODEL_ID,
    concurrency=4,
    request_count=100,
    prompt_input_tokens_mean=128,
    output_tokens_mean=128,
    streaming=False,
)

job = start_benchmark(
    endpoint=endpoint,
    workload=workload,
    role=ROLE,
    wait=False,
)

Amazon SageMaker AI sends the workload to the endpoint and collects performance metrics using NVIDIA AIPerf. We repeat the same benchmark for the G5, G6, and G7 endpoints.

For the initial comparison, we use non-streaming requests and evaluate:

• Output token throughput
• Request throughput
• Average request latency
• P50, P90, and P99 request latency

For interactive applications, you can also set streaming=True to measure time to first token (TTFT) and inter-token latency (ITL).

Step 3: Compare the benchmark results

The following table shows the results from running the same non-streaming workload across the three instance types.

For this specific workload and configuration, our testing showed that G7 delivered the highest throughput and lowest latency among the three configurations tested. At 391.3 output tokens per second, G7 provides approximately 60.8 percent higher throughput than G6 and 13.0 percent higher throughput than G5.

G7 also reduces average request latency by approximately 37.6 percent compared with G6 and 10.8 percent compared with G5. The difference is even greater at P99, where G7 reduces latency by approximately 54.7 percent compared with G6 and 20.2 percent compared with G5.

Evaluate streaming performance

We also run a streaming benchmark on the G7 endpoint to evaluate responsiveness for interactive coding applications.

G7 achieves a median TTFT of approximately 118 ms, allowing an interactive application to begin displaying generated content quickly. Its average ITL of 8.9 ms also provides consistent token delivery for use cases such as developer copilots and IDE-based code generation.

Key findings

The benchmark shows why measuring your actual model and workload is important when selecting inference infrastructure. For this Qwen3-Coder workload, ml.g7.12xlarge provides the strongest overall performance of the three tested configurations, delivering the highest throughput and lowest average and tail latency.

These results are specific to the model, serving configuration, token distribution, concurrency, and workload used in this benchmark. For production deployments, use workload characteristics that reflect your application’s expected traffic.

In the next use case, we take a different approach. Instead of choosing the endpoint configurations ourselves and benchmarking them, we use Amazon SageMaker AI Generative AI Inference Recommendations to explore candidate configurations and recommend the best options for the workload.

Use case 2: Generate deployment recommendations for Nemotron-3-Nano-30B

In this use case, we use Amazon SageMaker AI Generative AI Inference Recommendations to identify suitable deployment configurations for NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4. Unlike the benchmarking workflow, where we start with endpoints we want to compare, the recommendation workflow starts with the model, expected workload, candidate instances, and optimization objective. Amazon SageMaker AI evaluates viable configurations and returns measured performance recommendations.

For this example, we optimize for throughput using PerformanceTarget.THROUGHPUT. We also analyze cost per output token using the measured throughput and instance price, and review latency metrics such as TTFT and ITL. The accompanying notebook contains the complete implementation.

Step 1: Define representative workloads

We evaluate two workload profiles to understand how request shape affects the recommended deployment configuration.