From the early Kamakura period to the Muromachi period (1185-1573), this residence belonged to the Tō clan, who were the warlords the area centered around Yamato, Gujo. They built a castle on top of Mt. Shinowaki , which rises behind the garden, and they ruled the area for 320 years. The Tō clan excelled in both literary and military arts, and in the clan there was a particular talent for writing, with masters of waka (a style of traditional Japanese poetry that uses a "5-7-5-7-7" syllable pattern) appearing one after another for generations.