Robert had two children with his first wife (Elizabeth Fisher): Mary Elizabeth Scarborough and John Scarbury Scarborough. Robert had nine children with his second wife (Mary Pierson): Robert Scarbury Scarborough II, Rachel Keziah Scarbury, William Scarbury, James Scarbury, Isaac Scarbury, Sarah Scarbury, Susannah Scarbury, and David Scarbury.
Robert died on January 19, 1805 in Monroe County Virginia at the age of 96. He is buried in what is now West Virginia.
Robert is believed to have been a farmer who in 1737 sold 157 acres of his farm in Solebury Township, Bucks Coutny, PA. He had inherited this land from his father, John Scarborough II. He is believed to have then moved to 5 miles southwest of present New Market, Orange (now Shenandoah) County, Virginia. He had 400 acres on both sides of Smith's Creek. In 1737 he is believed to have moved to Winchester, Virginia and in 1747-1750 moved to Bedford County, Virginia.
Robert is believed to have been a Quaker.