AI GPU benchmarks

We finetuned the following models using LitGPT , each model was run using LoRA and configured with the optimal settings from the LitGPT config hub. The Alpaca 52K dataset was used for fine tuning. The charts display the:

• Time: total time taken in hours
• Price: total cost of the GPU time in USD

To compare inference latency and token throughput we used Ollama to serve LLMs and prompted the models via the REST API. These figures are not taken under load.

Each of the metrics are averaged across 817 prompts from the Truthful QA

• TTFW: Time to first word. This metric is measured from the time the request is sent to the API to the time the first JSON response is received. To use this metric in your own latency estimates you will need to add the network latency.
• TOK_PS: Tokens per Second. This metric is measured using Ollamas internal counters. It includes the prompt tokens and then prompt tokens evaluation time. It is calculated as:

  TOK_PS = (Prompt Tokens + Response Tokens) / (Load Time + Prompt Evaluation time + Response Evaluation time)

To compare performance of GPUs at common image processing tasks we used the Ultralytics YOLOv8 models in medium, large and extra large sizes.

Several model types were used; object detection, pose estimation and instance segmentation.

• yolov8m, yolov8l and yolov8x used the coco dataset.
• yolov8m-pose, yolov8l-pose and yolov8x-pose used the coco-pose dataset.
• yolov8m-seg, yolov8l-seg and yolov8x-seg used the coco dataset.