Which is where social hierarchy came in. For all the evils of serfdom et al, the modern world largely grew from the efforts of select nobles & upper classes who did not have to work to eat.
As screwed as the peasants were through all the ages, all the upper classes in the world won't make much progress if you don't have a relatively "free" peasantry from which to procure armies that can actually defend anything. Throughout history, when the free peasants (small farmers who could actually fend for themselves on land they lived on) became actual serfs (more or less slaves to the lords), their empires crumbled.
Until England worked this all out by creating the age of imperial commerce where the peasants became wage laborers who got paid before the harvest was taken. This is the creation story of the British Empire. The U.S. adopted it after Shays Rebellion.
The last unanswered question here: if all you have to participate in a market is to sell your body (brains and all), are you really free?
If working for money to eat is slavery, it is your stomach you are enslaved to. Empires are not the villain there. Were you free of the market, you would still be compelled to work to feed yourself.
You can even argue that the market is liberating, as it allows you to feed yourself with skilled trades.
That is what always bugs me with equating the need to work with slavery. Man has always needed to work, unless of course he lived off the backs of other working men, because shelter food and water have always taken work.