The problem with these devices are that they are often locked down, and even unlocked/hacked open getting them to run anything like mainline Linux is hard-ish.
A lot of them can be used just fine as a server with a fairly open OS. It's usually the GPU/camera/radios etc. that need special stuff, but things like USB (for networking) and flash work fine.
It's not even that hard to set up, go grab the kernel (some of qualcomm's kernels have these super crappy buildscripts that need python 2.x), build it, throw in busybox and dropbear, fastboot the image and you're good to go.
Android kills stuff when it feels like it wants more room to cache apps in ram. The background stuff like the media indexer takes CPU time and generates heat (these are pretty easy to overheat when used like this.) I think some of the older android userspace even has flaws exploitable over the network. etc.
