Dell R770 AI server: a practical buying guide

Which AI server is right for your workload?

An AI server brings the model, its working memory and the application into infrastructure you control. The useful starting point is a workload: a document assistant for a legal team, a vision pipeline in a factory, a speech service or a development environment for adapting an open model. Choose the model and expected user demand before choosing hardware. Plan application evaluation and data access alongside GPU memory, so the deployed service meets the needs of its users.

These Dell PowerEdge R770 configurations pair a refurbished enterprise rack server with one or two new NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Each card has 96 GB of GDDR7 ECC memory. The shared platform includes two 32-core Xeon processors, 256 GB system RAM, separate boot and data storage, 10 GbE networking and iDRAC10 Enterprise management. The distinction between the options is the number of GPUs, rather than a change to the underlying server specification.

The best AI server is the one that fits the chosen model, context length, concurrent requests, operating environment and support plan. These builds suit teams that need substantial GPU memory and an enterprise server platform. A smaller system may suit an occasional single-user assistant; a larger distributed cluster may be necessary for demanding model training. Compare the work the system must do rather than treating GPU count as a universal performance ranking.

What can you run on a private GPU server?

Private retrieval-augmented generation (RAG) combines an inference model with a searchable collection of your documents. The GPU can serve language models and embedding or reranking models, while the CPU, RAM and storage support ingestion, the vector database and application services. Retrieval quality, document permissions and evaluation matter as much as model size. Keep document permissions attached to retrieved material and filter results for the employee making each request.

For inference, GPU memory determines which model weights and working state fit. Useful workloads include internal assistants, classification, extraction and application APIs. Choose a serving engine that supports the precise model architecture and Blackwell GPU, then benchmark with representative prompts. Test latency at the expected number of simultaneous requests, not only a single short prompt. Long contexts, large responses and parallel users can change the memory requirement materially.

Parameter-efficient fine-tuning (PEFT), including adapter-based techniques, can adapt an existing model without updating every weight. It still needs memory for activations, training state and the chosen sequence length. A model that fits for inference may not fit for tuning. Establish the training framework, quantisation approach, batch size and dataset first. Full pre-training of a large foundation model is a different capacity problem; these configurations carry no promise that an arbitrary training job will fit.

Vision workloads include image classification, detection, segmentation and document understanding. Speech workloads include transcription, diarisation and speech generation where the software supports the relevant models. Simulation and GPU-accelerated engineering applications may also benefit when the application vendor supports this GPU and its licence permits the deployment. For each case, confirm software compatibility and measure accuracy or numerical requirements alongside speed. Review application and model licences, then agree a representative test for the intended workload.

Size model memory before you buy

Start with the model weights. As a rough sizing example, a dense 70-billion-parameter model at 16 bits per parameter needs about 140 GB in decimal units for weights alone: 70 billion multiplied by two bytes. At an idealised four bits per parameter the raw weight calculation is about 35 GB. These rough arithmetic estimates are a starting point; check the actual footprint of the selected model and runtime. Quantisation scales, metadata, padding and the implementation add to the raw figure.

Inference also needs memory for the key-value (KV) cache, temporary buffers, the serving engine and any additional models sharing the card. The KV cache grows with context length and concurrent sequences; its size depends on the model architecture and cache precision. An engine may reserve memory in advance. Leave headroom rather than sizing exactly to a weight-file download. A 96 GB GPU does not provide 96 GB entirely to model weights once the application is running.

System RAM and GPU VRAM are separate resources. The 256 GB of DDR5 helps the host run services, prepare data and hold CPU-side state; it does not turn a 96 GB card into a larger GPU. CPU offload can help some models load, but moving data across the host connection changes performance. Ask for a workload-specific sizing review using the exact model, quantisation, context length and concurrency target. Keep latency and throughput targets as acceptance criteria to measure rather than assumptions derived from memory capacity.

One GPU or two: capacity and practical limits

The one-GPU option provides 96 GB on a single card. It avoids the need to split one model across devices when that model and its runtime fit within the available memory. It can be a straightforward starting point for private inference, a RAG service or adapter experiments. Multiple services can share the GPU when their combined footprint and scheduling requirements permit, but isolated user-facing services may need explicit resource limits to avoid one job consuming another job’s capacity.

The two-GPU option provides 192 GB aggregate VRAM as two separate 96 GB devices. This is not an automatic unified memory pool. A multi-GPU serving or training framework must explicitly distribute a model or workload. Check the framework’s tensor or pipeline parallel support, the model implementation and how the cards communicate in the selected topology. Do not assume a particular interconnect or linear speed-up simply because a second GPU is installed.

A second card can also run a separate model or workload: for example, one service for generation and another for embeddings, vision or development. That separation may be more useful than splitting one model. Compare independent jobs against a distributed job in a representative proof of concept. More GPU capacity also increases power and cooling requirements, so the decision belongs in the facilities plan as well as the model plan. Both options retain the same CPUs, host memory and storage listed below.

Dell’s R770 GPU requirements matter

Dell’s R770 GPU specifications list support for up to two NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB cards at a maximum of 450 W each in this platform. That supported R770 power limit is the relevant basis for this offer. Use this platform power limit when planning the server’s electrical and cooling requirements. The exact behaviour still depends on the application, driver, firmware and operating conditions.

The Dell guide identifies riser configurations 6-2 and 11-2, HPR Platinum fans and iDRAC firmware version 1.20.60.55 or later for this GPU configuration. It specifies Ubuntu 24.04.02 with kernel 6.11 or later in the documented software requirements. Use that documented combination as the starting point for the software deployment plan. Keep the operating system, NVIDIA driver, CUDA runtime and application framework on a validated combination.

Power cabling is platform-specific. Dell identifies DCNHT for slot 2 and K8FW0 for slot 7, using the extended-soldering versions required for RTX PRO 6000. Include the risers, airflow, firmware and correct GPU power-cable versions in the written configuration and acceptance checklist. Check Dell part numbers and cable versions against that checklist when accepting the final configuration.

Plan rack space, power, heat and connectivity

A rack server belongs in a suitable equipment room or data centre. Check cabinet depth, rail fit, rear cable clearance, airflow and the route into the building before ordering. Confirm that the delivery team can reach the room and that the rack can support the installed load. Verify the included rack kit against your cabinet’s mounting arrangement. Allow access for maintenance and safe handling rather than filling every remaining space around the chassis.

The configuration has two 3,200 W power supplies. That nameplate rating is not a measurement of continuous consumption and does not mean the server always draws 6,400 W. Size circuits, PDUs and UPS equipment using the supported electrical input requirements, redundancy arrangement and an appropriate worst-case load. The two GPUs alone have a combined platform limit of up to 900 W in the two-card build; processors, memory, storage, fans and conversion losses add to the system load.

Almost all electrical energy used by the server becomes heat in the room. For a clearly labelled planning example, a measured 1,500 W load would release roughly 5,100 BTU per hour, using about 3.412 BTU per hour per watt. That is an illustrative calculation, not a measured R770 result. Have the facilities team check cooling capacity at expected ambient temperatures, airflow direction and backup power. Enterprise GPU servers can be loud; an ordinary shared office is rarely the right installation location.

Dual-port 10 GbE RJ45 is included. Check switch port capacity, copper cabling, uplink contention and the size of datasets moving to the server. A faster GPU does not accelerate a saturated storage or network path. Segment iDRAC management from user traffic, restrict remote administration and plan patching, monitoring, backups and recovery. RAID policy and usable capacity depend on configuration; keep a separate recovery copy and test restores.

Can you run your own AI server?

Yes, provided you have an appropriate location and someone responsible for operations. Ownership gives you control of the hardware and deployment choices. It also gives you responsibility for security, availability and software maintenance. Decide who approves model changes, who monitors the service and who can restore it after a failed update. Assign an application owner and an operations contact, with a clear route for users to report problems.

For sensitive data, map where inputs, retrieved documents, logs and outputs travel. A locally hosted model can still call an external tool or telemetry endpoint if its application is configured that way. Review model licences, retention settings and access controls alongside South African data-protection obligations. Separate development and production credentials, protect model endpoints and record operational changes. Start with an internal pilot and a small evaluation set before allowing the system to act on business records.

Compare ownership with hosted GPU capacity over a realistic utilisation period. Include electricity, cooling, rack space, backups, software, staff time and maintenance rather than comparing only a purchase price with an hourly rental rate. Stable, regularly used workloads can make ownership attractive; occasional experimentation has different economics. Build a business case from your own expected usage and the actual cost of the hosting alternative.