You can't limit the amount of RAM, but the RAM metrics are somewhat deceptive; it uses memory-mapped files, so your virt usage is going to basically be the size of your whole dataset. Your resident usage is going to consume up to as much RAM as isn't being used by other processes on the machine; the kernel's VMM should manage how much resident memory is actually used.

In practice, you generally want your DB server(s) on their own machines if RAM is ever possibly going to be a concern. Any database that constantly has to hit the disk isn't going to perform worth a hill of beans.