Maybe in terms of the income tax that we know today, but if you did any importing or exporting you paid a heck of a lot of taxes via tariffs. If you ever get a chance to read up on the pre-Civil War period you'll find that the south was responsible for about 87% of total Federal revenue and almost entirely due to tariffs.
That's an excellent question ... my experience over several years helping campaign for a party that wanted to eliminate progressive taxation was that:
* people are terrified of fundamental changes to systems that provide their 'benefits', even if the existing system involves literal slavery
* people get really, really angry when you mention the 'slavery' thing; they don't seem to like really thinking about how the sausage is made, so to speak
* slavery is easy when quality of life is so good; it's easy to get bent out of shape when 40% of your income is being taken when your income is diddley-squat; when you're living like a 1700s king already, meh
* socialists have entirely and completely conflated the concept of a just society with compulsory taxation and state welfarism that it is almost impossible to get people to even acknowledge that the two can be considered separately
* self-described capitalists are often only capitalists when it suits them, otherwise they are often happy to suckle from the State tit to a far more egregious extent than any 'welfare queen' ever did
It's an interesting subject.