Poor understanding of / fluency with units is definitely one of the things that sets people back hardest when they get to higher levels.

I say start throwing units in as early as possible (grade school?).