Technically correct, however the brake booster only activates when you are braking, in which case the engine is either spinning down to idle or idling. I am not certain of this, but I think this means it is essentially "waste work", because the engine would be idling anyway, and the power drain of the brake booster is not significant enough to warrant feeding the engine extra gasoline.

This is distinct from things like A/C, which place a non-trivial amount of load on the engine even at idle, and often require things like a higher idle speed while engaged.