The mansion was built by Stephen Salisbury, a general goods store owner who became an influential local leader in Worcester. His son, Stephen Salisbury II, was a selectman for Worcester and then was elected a State Representative and State Senator. He also was a founder of institutions such as the American Antiquarian Society – which is now the oldest historical society in the United States – and Worcester Polytechnic Institute. His son Stephen Salisbury III was also a State Senator and founded the Worcester Art Museum.
The mansion is now owned and operated by the Museum of Worcester, which offers tours to the public.