Matthew Vassar’s niece, Lydia Booth, who Matthew Vassar said "more than any other, and more than all others" accounted for his decision to found Vassar College, died suddenly, at the age of 51. The stepdaughter of Vassar's sister Maria, Lydia Booth opened the Poughkeepsie Female Seminary in 1837 and later, with her uncle's help, established Cottage Hill Seminary on Garden Street in Poughkeepsie. Although her death occurred while her uncle was still deciding among several philanthropic possibilities, her influence caused Vassar's eminent Professor of History Lucy Maynard Salmon to call her "the real founder of Vassar College."

As his college was becoming a reality, the Founder acknowledged his debt to his niece in an address to the trustees on February 23, 1864, saying:"It is due to truth to say that my great interest on the subject of female education was awakened not less than twenty years ago by an intimate female friend and relative, now deceased, who conducted a seminary of long standing and character in this city . . . It was this fact, more than any other, and more than all others, that awakened me early to the possibility and necessity of an institution like the one we now propose."