Congress Gardens is in the heart of the Cobble Hill Historic District, surrounded by Cobble Hill Park, the Verandah Place mews, a Gothic Revival church, and 19th and 20th century townhouses.

Both sides of Congress Street between Court and Clinton Streets were originally part of the farm of Cornelius Heeney (1754-1848), who gave his land to the Roman Catholic Church. The church built a new parish, rectory, and orphanage on the south side of the street in the 1830s. Congress Gardens was built by Lama, Proskauer & Prober architects in 1949 on the site where the orphanage once stood. The three adjoining six-story apartment buildings were clad in brick and surrounded by lawns to complement the surrounding townhouses and gardens.

Not counting apartments that have been combined, there are 114 apartments in total: the 200 and 210 Congress Street buildings each have 36 apartments and the 220 Congress Street building has 42. The complex, a Class A multiple dwelling of fireproof construction, was converted from a rental to an apartment corporation in 1981.