Not exactly on the deep learning side, but I am a scientist who does some computatinoal work on the GPU.
You're right that we don't use our computer card for our graphics and, as many people pointed out, the computation will wind up running on a headless server. On the other hand, during development, our code will be running on our desktops and will be using the GPUs on the graphics card. I've witnessed a developer's frustration with X11 driver issues cause a project to transition from CUDA to OpenCL.
Granted, he didn't just throw a hissy fit an port the code. Rather, it inspired him to write up a tradeoff analysis of OpenCL versus CUDA that showed that OpenCL offered this specific project significant benefits and that the main selling points of CUDA weren't being used in this instance. He also wound up rage coding a prototype port to OpenCL to go along with the proposal.
The project would have moved eventually, since it was the better engineering decision in this case, but it was about a half decade away on the timeline until the lead dev's compositor stopped work.
I can only speak from my own experience (2-3 GPU rigs running tensorflow) but I have never had any trouble. Full Ubuntu desktop. Training models on the GPU does not cause any problems with display lockups etc.
