The public library in Vassar was established in 1906, when Mrs. Emma Bullard willed her home and property to the Village of Vassar to be "used as a public library forever." The Bullard Library operated in the former Bullard home for 56 years. As part of their compensation, librarians were permitted to live rent-free in the upper level of the home, while the library was operated on the lower floor. When the Village of Vassar became a city in 1947, the Library became a city chartered public library. In 1962, with additional funds from the Anely Sanford Memorial Trust, a new building was erected on the site of the former home.

With a steady growth in collection and usage by area residents, the Library Board made a concerted effort to improve the quality of services offered to the community. In 1976, the Board hired the first professional, master-degreed librarian, David Hart. The growth in community usage of the library necessitated three additions to the Library in 20 years. In 1976 a conference room and improved public rest rooms were added to the original 1962 building. An addition that doubled the floor space was added in 1986. In 1996, a third, $250,000 addition that created a new conference room, an improved children's area, as well as a reconfiguration of the entire library was completed entirely with donated funds and grants.

Since it's beginning, the Library operated as a village library, and later as a city library, with contracted services to the surrounding townships. In 1996, a planning committee composed of city, township, and library representatives recommended the formation of a "district library". The City of Vassar, Tuscola Township, and Vassar Township approved of the district library agreement, and later that year the voters of all three municipalities approved of a 1-mill tax levy exclusively for the newly formed district library. In 1998, the Juniata Township Board approved of joining the district library, and later that year, the voters of the township who do not live in the Caro School District (which is part of the Caro District Library) approved of a 1-mill tax levy for the district library.

Currently, the Bullard Sanford Memorial Library is a municipal entity onto itself, and is governed by a Library Board composed of representatives of the four municipalities that fund the district library.