"I'm an interactive shell so I expect to have a lot of tiny packets" is what the delay is for. If you want to turn it off for those, you should turn it off for everything.
(If you're worried about programs that buffer badly, then you could compensate with a 1ms delay. But not this round trip stuff.)
(If you're worried about programs that buffer badly, then you could compensate with a 1ms delay. But not this round trip stuff.)