Customer will not, and will not allow third parties to: (i) use these Services
to create, train, or improve (directly or indirectly) a similar or competing
product or service or (ii) integrate these Services with any applications for
any embedded devices such as cars, TVs, appliances, or speakers without Google's
prior written permission. These Services can only be integrated with
applications for the following personal computing devices: smartphones, tablets,
laptops, and desktops. In addition to any other available remedies, Google may
immediately suspend or terminate Customer's use of these Services based on any
suspected violation of these terms, and violation of these terms is deemed
violation of Google's Intellectual Property Rights. Customer will provide Google
with any assistance Google requests to reasonably confirm compliance with these
terms (including interviews with Customer employees and inspection of Customer
source code, model training data, and engineering documentation). These terms
will survive termination or expiration of the Agreement.
This is the TOS specifically for two of the managed services - it says so right in the link "Google Cloud Machine Learning Group and Google Cloud Machine Learning Engine"
These specific ML-as-a-service products are pretty easy to plug into and resell to third parties, which is why the wording is so restrictive in the TOS.
Many of Google Clouds other ML products, which typically are more customizeable and powerful, do not have these restrictions. The way your comment is worded makes it seem like this restriction is on all of Google Clouds ML products, but it's not.
I don’t understand these restrictions. Why is reselling bad? I would think people would still prefer first party services and even if they don’t you still get utilization in any case. Sure, there can be some outlier bad cases but these kind of restrictions can be a complete turnoff for enterprise users who just don’t want to take any legal risk of breaking TOS.
It might have not been clear from my comment, but I was specifically referring to those ML as a service. The TPUs seem to fall under GCP which has a less restrictive ToS.
I do understand they don’t want to make it too easy to resell their research. But having access to all your source code, training data and documentation is a bit extreme. Why can’t they just tell you to stop selling your product?
12.1 The following terms apply only to current and future Google Cloud Platform Machine Learning Services specifically listed in the "Google Cloud Platform Machine Learning Services Group" category on the Google Cloud Platform Services Summary page
- Cloud AutoML
- Cloud Text-to-Speech
- Dialogflow Enterprise Edition
- Google Cloud Data Labeling
- Google Cloud Natural Language
- Google Cloud Speech-to-Text
- Google Cloud Video Intelligence
- Google Cloud Vision
.. We will force you to admit that we OWN machine learning for all consumer use cases and will happily sue you just for renting our servers, or force you to turn over your Intellectual Property Rights to us. Thanks.
A common case is an employment contract that has an NDA or non-compete or non-poaching section survive the employment itself. I can't say, "I quit, therefore my employment contract doesn't apply any more, therefore I'm going to push all the company secrets to GitHub and you can't stop me."
To be fair, "Warranty Void if Removed" stickers are common but in many cases have no legal standing both in the US, EU, etc. They're just a common "boilerplate" kinda item.
>(ii) integrate these Services with any applications for
any embedded devices such as cars, TVs, appliances, or speakers without Google's prior written permission.
Genuinely curious about the restriction regarding embedded devices.
If there's a mishap, it is the responsibility of the owning company, certainly not Google.
If they have patents for technology applied to those items, they would lose the patents if the didn’t enforce them. I’m sure they could arrange a license if you had a use case, and that may be free. Again, if they demonstrate they don’t enforce their IP (which is possibly incredibly valuable) at least in writing they could lose all rights to enforce their patent. I’m no IP lawyer but I paid one a bunch of money once.
Eh, you’re right, I was confusing it with trademarks. I’ll dust off that pitchfork, there isn’t any good excuse for them naming the industries they would like to own and barring their customers from competing with them.
This doesn't seem plausible. They have tens of thousands of patents on all sorts of stuff, and people "infringe" on them all day long with no repercussions. Some of those patents will probably be violated when I click the "reply" button under this message.
I was taken aback by this one as well. Basically this is an ineffectual attempt at preventing giving away their AI advantage by letting others use their proprietary models (and thus, indirectly, the gigantic dataset that make those models so good) as a "teacher" for their own models, using the fairly standard model refinement techniques popularized by their own Geoff Hinton. Keeping the cards close to their vest, as it were.
See e.g. https://www.quora.com/What-is-hiybbprqag for an example where Microsoft trained their ranker on Google's long tail and substantially reduced the relevance gap on the cheap, and in a way that makes it impossible for Google to trivially regain the advantage. People misinterpreted this as "Microsoft is copying the results", but that's not what it was. Google was unwillingly teaching Bing how to rank. Google responded with personalized search.
I don't think they can realistically prevent this, though. And it's not going to be anywhere near as traceable as "hiybbprqag" was.
Google has been using different trick for the same reason with their own employees: that famous 20% of working time that can be dedicated to personal projects. You might think they did that to increase employee satisfaction, but the real reason is that this way they own copyright to whatever their employees create in their spare time, and avoid Facebook-WhatsApp scenario (i.e. having to spent $14B on something you could have aquired for free).
Now I guess they apply the same principle to their GCP customers.
>but the real reason is that this way they own copyright to whatever their employees create in their spare time
Yeah it's not their spare time; it's work time. I would enjoy being able to spend one day a week working on whatever I like _for the company that employs me_. It's not a vacation day you spend in the office.
It was never for personal projects. The actual IPO quote is "We encourage our employees, in addition to their regular projects, to spend 20% of their time working on what they think will most benefit Google".
It's often done in other companies to a lesser degree with hackdays/weeks and is a good way to clean up tech debt or kickstart new product ideas.
Cloud TPU pods are seriously amazing. I'm a researcher at Google working on speech synthesis, and they allow me to flexibly trade off resource usage vs. time to results with nearly linear scaling due to the insanely fast interconnect. TPUs are already fast (non-pods, i.e. 8 TPU cores are 10x faster for my task than 8 V100s) but having pods open up new possibilities I couldn't build easily with GPUs. As a silly example, I can easily train on a batch size of 16k (typical batch size on one GPU is 32) if I want to by using one of the larger pod sizes, and it's about as fast as my usual batch size as long as the batch size per TPU core stays constant. Getting TPU pod quota was easily the single biggest productivity speedup my team has ever had.
Are TPUs drop-in replacement for CUDA if you were using TF?Can I simply change device from CUDA to TPU and run any TF code? Last I heard, TPUs still had long way to go towards making this happen...
IIUC, the difference is that the TPU allows insane data parallelism - exactly the giant batch sizes that rryan mentions. Google carried out extensive research on batch size, demonstrating that you can greatly increase training efficiency with giant batches. (And with tweaks to optimizers or some regularization, can push the "efficient" batch size domain even further.)
The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.
This manifests in the training time + cost for imagenet in the stanford DAWN benchmark. The (partial) TPU pods win on training time, with a clear exponential speedup as the pod fraction increases. (How do you use more cores? Increase the batch size.)
I’m looking at Imagenet training time for Resnet-50 [1] and it appears that 32 V100 chips are faster than half of a TPUv2 pod (64 chips). Are we looking at different benchmarks?
If you want to view their header image at larger size but right-clicking doesn't give you the option to "Open image in new tab", the direct link is below. Not a big deal, but it might save a few clicks for some:
Nvidia could have released a DL specific chip a long time ago, if they wanted to. I’m not sure why they haven’t (market not big enough?), but they probably will at some point.
They release a "data center" specific version of their gpu with slightly improved stats compared to the consumer models, and 10x the price... (And include a "no data centers" clause in the consumer model terms of use.)
(I work at Google on compilers for ml, including compilers for Nvidia gpus.)
Devices like the v100 and t4 are ml-specific chips. You can do graphics stuff on them, but that doesn't mean that Nvidia if leaving a ton of ml performance on the table by including that capability. Indeed there may be economies of scale for them in having fewer architectures to support.
V100 has 640 tensor cores, and 5k general FP32/64 cores. Most of DL computation is done by tensor cores. Can you imagine how much faster it would get if they released a chip with say 10k tensor cores?
> Can you imagine how much faster it would get if they released a chip with say 10k tensor cores?
I can, actually. :) Adding 10k tensor cores to a GPU would not make it run much faster, and would be prohibitive in terms of die space. Moreover getting rid of the 5,000 FP32 cores would slow down DL workloads significantly.
The 640 Tensor Cores vs 5,000 F32 cores comparison is misleading, because they are not measuring the same thing.
An "FP32/64 core" corresponds to a functional unit on the GPU streaming multiprocessor (SM) which is capable of doing one scalar FP32 operation. One FLOP, or maybe two if you are doing an FMA. V100 has 5120 FP32 units and 2560 FP64 units.
In contrast, a "Tensor Core" corresponds to a functional unit on the SM which is capable of doing 64 FMAs per clock. That is, a Tensor Core does 64 times as much work as an FP32 core. Integrated circuits aren't magic, if you're doing 64 times as much work, you need more die space.
Moreover, there is nothing to say that nvidia isn't able to use some of the same circuits for both the fp32 and tensor core operations. If they are able to do this (I expect they are) then reducing one does not necessarily make space for the other.
Increasing the number of tensor core flops by a factor of 20 (500 -> 10,000) would not make the GPU 20x faster, probably not even 2x faster. This is because you will quickly run into GPU memory bandwidth limitations. This is not a simple problem to solve, nvidia GPUs are already pushing what is possible with HBM.
Lastly, although you're correct that, in terms of number of flops, most DL computation is done by tensor cores (if you've written your application in fp16), that doesn't mean we could get rid of the f32 compute units, or even that significantly reducing their number would have minimal effect on our models. Recall Amdahl's law. We usually think about it in terms of speedups, but it applies equally well in terms of slowdowns. If even 10% of our time is spent doing f32 compute, and we make it 10x slower...well, you can do the math. https://en.wikipedia.org/wiki/Amdahl%27s_law
Indeed, I was just looking at an fp16 tensor-core cudnn kernel yesterday, and even it did a significant amount of fp32 compute.
The implicit argument I read in parent post is that nvidia could build a significantly better DL chip "simply" by changing the quantities of different functional units on the GPU. This is predicated on nvidia being quite bad at their core competency of designing hardware, despite their being the market leader in DL hardware. It's kind of staggering to me how quickly nonexperts jump to this conclusion.
I think your main point is that memory bandwidth would prevent the performance speedup. Are V100s memory bound when executing F16 ops on tensor cores?
Second, do we really need dedicated FP32 cores for DL? Tensor cores accumulate in FP32 (is that what you meant when you said they did a significant amount of FP32 compute?), and recent papers indicate we’re moving towards 8 bit training [1]. Besides, do TPUs use dedicated FP32 hw?
Finally, if the memory bandwidth is indeed the bottleneck, perhaps all that die area from FP32 and especially FP64 cores could be used for massive amount of cache.
V100s are often memory bound when using tensor cores, yes. But I guess my point is broader than that. There is a "right shape" for hardware that wants to excel at a particular workload, depending on the arithmetic intensity, degree of temporal locality, and so on. The point is that you usually can't just turn up one dimension to eleven, it's not usually that simple.
For example, massively increasing the GPU last level cache size would not have the effect of increasing memory bw much on most workloads, because cache only helps when you have temporal locality and gpus like to stream through many GB of data.
This is covered in Hennessy and Patterson if you're curious to learn more. I also talk about it some in the video I linked above.
(Also I doubt that getting rid of f64 support would be a significant die size win. I notice that v100 has, in their marketing speak, twice the fp32 cores as fp64 cores. What do you think are the chances that Nvidia decided a priori this is the optimal ratio? What if instead they are sharing resources between these functional units, at a ratio of two to one?)
To the question of, do you really need fp32 cores, I am not aware of any "widely deployed" GPU model today that does not do significant fp32 work. Perhaps there is research which suggests this isn't necessary! But that is a different thing than we were talking about here, that Nvidia could somehow make a much better chip for the things people are doing today.
I don't want to speak to the question of whether TPUs have f32 hardware, because I'm afraid of saying something that might not be public. But I think the answer to your question can easily be found by some searching and is probably even in the public docs.
I’d really like to know this as well. NVidia has became very AI centric company but this has been a huge blind spot. GPUs with power and chip real estate wasted on rendering hardware are unnecessary and relics of legacy for deep learning. Why they haven’t yet designed CUDA++ only chip yet?
Probably because most of their customers don’t work with Google-like problems, and will not buy chips that are 10x faster on paper but 10 times slower on the problems they DO have, at 10x the price...
Unfortunately for Google, NVIDIA's offerings are very strong, and TPUs are a pain the rear to use and require TensorFlow, which in itself is a pain to use, making it doubly painful, to the extent that using their offering requires a significant degree of desperation or not knowing any better.
If it's "available" but doesn't do anything useful, it's IMO not really "available". It's a good start though. I hope they don't throttle it in favor of TF because that'd only prolong TF's agony.
Yes I used Keras before PyTorch came out. IMO, Google needs to start over rather than try extensive wart removal on something that consists almost entirely of warts. These days Keras has subpar multi-GPU support (two GPUs are often slower than one), less flexibility, and more un-debuggable magic than PyTorch. And it doesn't look like things will be improving much (if at all) with TF 2.0.
And the TPU is a story of its own when it comes to usability. Take a look at their docs. TPU actually has to run on a separate machine, which you need to provision manually using their `ctpu` tool, and training of a trivial network on CIFAR looks like this: https://github.com/tensorflow/tpu/blob/master/models/experim...
If you're just using off-the-GitHub models and pressing them into service, sure, Keras is adequate. If your work actually requires research and experimentation (as most practical problems on which actual cash money can be made do) you'll be much better served with PyTorch. The main cost in all this is not actually implementing or training your final model. That can be done in any framework. The cost is finding a model that works really well for a particular task. You will get there a lot sooner and with a lot fewer headaches if you use PyTorch for this more "researchy" part.
As to deployment, once you figure out what works, there's also libtorch, or if that doesn't work for you for whatever reason, implementing a model you know working hyperparameters for can be done in a day or two on whatever framework works for your backend, including TF.
A snippet from https://cloud.google.com/terms/service-terms#12-google-cloud...: