If I want to run an open weight model on my local server the general assumption is that I need GPUs with enough ram to fit the model. Nvidia offers various GPUs in the RTX series and there are a few other options as well. I was pondering whether I can use regular DDR4 or DDR5 ram on a server instead. DDR ram is much cheaper than GPUs and in a corporate setting one may already have a server with say 0.5 TB or 1 TB DDR ram already standing around for other uses. That would allow running some fairly large models. My understanding is that the key difference between GPUs and DDR ram is the access speed, with values around 25-100 GB/s for DDR ram versus 936 GB/s or 1792 GB/s for RTX GPUs. Is that the major difference and does a factor 20 in access speed translate to roughly a factor of 20 in AI model response time? On GPUs for most common queries you get a response in a few seconds so if I'm willing to way for a couple of minutes instead is that the only drawback for switching to cheaper DDR ram?

Full article content could not be extracted automatically. Read the original below.