In order to give money to one group, you have to take money away from another group
This is not true. The government can credit anyone’s bank account for however much it desires. This is how federal spending already works; it is not dependent on tax revenue.
Perhaps there are many more tragedies that should have been made right which were not. That does not mean that we cannot identify, quantify and rectify other injustices.
Reparations could take many forms including infrastructure, education, etc. It doesn’t have to be a direct deposit into the checking account of the descendants of slaves. However, this is the most direct way to do it.