As for Grant, according to this source he freed his slave prior to the war: http://www.factcheck.org/2007/12/presidents-who-owned-slaves/
Grant owned a slave named William Jones, whom he freed in 1859. Between 1854 and 1859 Grant worked and lived on an 850-acre farm in Missouri, near St. Louis, that was owned by his father-in-law. Grant’s wife, Julia, also owned slaves, and during Grant’s management of the farm he worked along with one of them, a man named Dan. The farm is now the Ulysses S. Grant National Historic Site, part of the National Park Service.
Another source: https://pastexplore.wordpress.com/2015/06/29/did-ulysses-s-grant-own-slaves-during-the-civil-war/
There are literally no other pieces of historical evidence to suggest that Grant ever owned slaves at any point after 1859. The quote about Grant not being able to find any good labor is a complete fabrication and you will not find it in his edited papers or any newspapers from the time. It’s simply not true.
Grant wasn't exactly prosperous, lol.
As far as Lee, he never owned any slaves. His wife inherited the Custis estates and the slaves attached to them. Her father's will stipulated that his slaves be freed within five years of his death. George Washington Parke Custis died in 1857. The Custis slaves were freed in 1862 in accordance with the will. Lee never owned any slaves to free.