Wage labor allows a higher rate of surplus value extraction by turning not just bodies, but labor power itself into a commodity. It also absolves employers of the need to feed, house, and pay for sleeping and non-work time of the person, now only needing to account for their active working hours.

Discovering this seemed to come along naturally with industrialization and more mobile capital requiring more mobile labor. Its part of the stable transition from feudalism -> capitalism, mostly starting in Britain and spreading to the rest of western europe.

it wasn’t. it was reformed into the prison-industrial complex

Abolished? Or rather optimized? Slavery was just an economical outdated method of production. Like why we don’t smelter iron in clay furnaces anymore. Also enslaved peoples weren’t consumers, since capital requires more markets, it was beneficiary to make them into consumers.

Enslaved people are also too hard to control. The best form of control is where the person controlled thinks they are free.

In addition to the answers already here, Southern slave plantation owners controlled the largest swathes of prime agricultural and cotton growing farmland. Almost the only cost of maintaining a slave is food and a little clothing, so as an industry the plantations were essentially self-sufficient just from their own produce. This meant that, as they were not paying money for food and clothing, they could keep prices for food and clothing high, draining money out of the industrializing North.

Capitalists, slightly contrary to other answers here, do actually need to pay their workers - at minimum - enough to reproduce their labor (at least until the extreme short-termism of neoliberalism arises) so high prices for food and clothing means paying workers more, which means setting higher prices for goods to keep the same profit margin, which means being outcompeted in the world commodities market by countries in which food is cheaper.

It’s also impossible to mechanize farming on a slave plantation, for the obvious reason that people forced into slavery will gladly sabotage your expensive machines. So, unlike capitalist firms, plantations are not in a race for automation to cut labor costs and outcompete each other - each plantation simply produces as much food as it has land available, which has the effect of preventing the rate of profit for farming as a whole from falling.

To explain: in each particular capitalist field of industry, each firm within it is trying to undercut the others by lowering its prices, which (if it wants to also make profit to spend on future expansion) it can only do by lowering its costs. And as all the firms are buying their input materials on the same market for the same price, the only way to lower costs is to spend less on paying workers - whether by reducing their wages, making them work more for the same wage, or, most significantly for the long term development of capitalism, by replacing them with machines.

Each firm is therefore in a race against the others to automate away as much of the workforce as they possibly can, with steam engines, spinning jennies and every other form of industrial machine - but as we know, profit can only come from the surplus value created by workers, meaning that, as the number of workers in an industry undergoing automation falls, so does the overall rate of profit for that industry.

However, in contrast, farming as a slave-based industry is able to maintain ‘ownership’ (in every sense) of a far larger fraction of the total number of workers in society - and therefore a much larger fraction of the total surplus value created by society - than farming under capitalism, meaning plantation owners as a class have much greater wealth, relative to industrialists, than they would as owners of farms under capitalism.

But of course, this situation harms the Northern industrialists, because high food prices mean they can’t compete on the world market. The logic of capitalism - the logic of the current level of development of the means of production - demands that farming be subject to mechanization. Slavery is outmoded, not because society has reached a higher level of moral and ethical development, but because it interferes with the seeking of maximum profit.

For as long as industries were competing with each other on the same continent, all suffering under the same prices for food, slavery as the primary basis for agriculture could carry on - but as soon as capitalist firms have eaten their local competition and started seeking out new markets on foreign shores, slave-agriculture becomes a fetter on accumulation that Capital must abolish. Under other circumstances, capitalist farms would simply appear and undercut the prices demanded by the plantation owners, but in the US the monopoly that the Southern slavers had over huge tracts of much higher quality farmland in a much better climate made this impossible. So, in the face of these totally incompatible material interests, the only possible outcome was the violent overthrow of the regressive class and the dissolution of their obsolete mode of production.

Because of the invention of the steam engine. It’s no surprise the abolitionist movement was most successful in England, where steam engines first got their proper use, both as locomotives and boats. In the US, it is finally abolished country-wide after the industrialised North beat the South.

Slavery in general or a specific period?

The whole transition from pre-capitalist modes of production (whatever they may have been) to the capitalist mode of production happened because of the mechanization of agriculture and textile making. These lowered the value of labor-power (which is the cost of reproduction workers, the 2 biggest ones being food and clothing) and allowed for the generation of massive amounts of surplus value on a historical scale. Wherever the mechanization of agriculture and textiles occurred, the capitalists gained power over the other proprietor classes. They leveraged this power to further expand the mechanization of agriculture, and thus the capitalist mode of production wherever they could.

In fact, this process of expanding capitalism still is ongoing today. There are something like 2 billion "small holding farmers who interact with the market to some extent, but are not practicing capitalist farming (socialized, mechanized commodity farming). This is why capitalism today, which has assumed a world dimension has still not run into its final limits

This isn’t meant to be “the” answer, just one part of whole.

The major fault line in the antebellum period wasn’t allow slavery / abolish slavery, it was whether or not to allow slavery in the “territories” i.e. the western parts of the US that were claimed by the US but not yet considered states. Well, the southern slavers did worry about the wholesale abolition of slavery, but that’s because they were always paranoid about that. But for the southern slavers, what happened in the territories was critical because 1.) if more states are created that do not allow slavery, eventually the southern states would be outvoted in the senate and lose their disproportionate power since the senate was created in part as a way to keep the southerners happy. But also 2.) these southern slavers were making huge profits, and those profits needed to be reinvested. These slavers looked at the western territories and had dreams of using their profits to buy land and recreate the plantation system all across the west. Just because slavery was the mode of production, these southern plantations were creating tremendous amounts of capital. Where under capitalism, you might reinvest that capital in better equipment to be more productive with less labor, slavers had no such incentive because their slave labor was so cheap. But you still have to reinvest your capital (the southern economy was, after all, not a pure slavery system and it had plenty of elements of capitalism in it just by being part of a larger capitalist economy). Thus they were desperate to open up the west to slavery.

Up in the north, though, I don’t think too many people really were focused on abolishing slavery in the south. The idea that the war was fought to abolish slavery is a myth (that’s different from saying the war was about slavery, which it was). You had three slave states join the union cause, the emancipation proclamation wasn’t declared until well into the war, and Lincoln himself I believe offered to let the slave states keep slavery forever if they stayed in the union (which again, considering what I said above, wasn’t much of a consolation prize to the slavers who needed to be able to expand slavery into the territories). None of that is consistent with the union wanting to abolish slavery nationwide. Instead, what the northerners cared about was “Free Soil”. The Free Soil Movement is critical to understand in order to understand the source of the conflict. We all know that America was founded on stealing indigenous land and using that as a “release valve” on class conflict. As we approach the civil war, the amount of land available for the taking was starting to get low. Everyone knew there would be a massive amount of genocide, ethnic cleansing, and then eventually new lands for further settler colonialism. But if slavery was allowed in these newly stolen lands, everyone knew that there was no way the yeoman farmer could compete with the plantation system. Not only that, but both the existing capital stocks as well as the super profits generated by the plantation system meant that the slavers could potentially buy up vastly disproportionate amounts of land, squeezing out the would-be yeoman farmers and capitalist farmers for that matter. The Free Soil Movement was the idea the idea that slavery should not be allowed in the territories, not because of moral indignation, but because both these would-be farmers as well as capitalist investors wanted it all for themselves.

So ultimately, I think a lot of the battle over the abolition of slavery came down to a power struggle between northern capitalism (with an understanding that they needed to keep the masses in north happy with the prospect of "free real estate) and southern slavery. Abolishment came as a way to break the political, social, and economic power of the slavers. Not dissimilar to how the emerging bourgeoisie in Europe had to overthrown the political, social, and economic primacy of the aristocrats. In Europe, the defeat of the aristocrats didn’t mean they lost their wealth and power, just that they had to be subordinated. And likewise in the US, the plantation owners by and large still did fine for themselves after slavery was abolished; they just no longer were allowed to share power more or less equally with the industrial capitalists.