The original Gardiner reportedly purchased the island in 1639 from the Montaukett Indians for "a large black dog, some powder and shot, and a few Dutch blankets." The indians called the island Manchonake, while the Gardiners initially called it Isle of Wight. The original 1639 royal patent gave Gardiner the "right to possess the land forever". The island was granted a new patent in 1665 by Governor Richard Nicolls, and annexed by East Hampton in 1688. Gardiner established a plantation on the island for growing corn, wheat, fruit, and tobacco, as well for raising livestock. The island was later the scene of wars between Native American tribes during the Pequot War and was reportedly visited by Captain Kidd. A popular legend holds that Kidd buried treasure there. The island was used by the British to store provisions during the American Revolutionary War and the War of 1812.
From 1854 to 1894, the island was the site of the Gardiners Island Lighthouse. Most proprietors lived in the island up until 1920, when it was leased and became a hunting preserve. During World War II it served as a military base with an airfield. Owing to the high cost of upkeep, in 1937 the island was put up for sale but was bought at the last minute by a relative, Sarah Diodati Gardiner, for $400,000. Upon her death in 1953, the island passed to her nephew Robert David Lion Gardiner, and his sister, Alexandra Gardiner Creel. Their aunt had also set aside a trust fund for upkeep of the island, but it was exhausted by the 1970s.
The island was the subject of a family dispute regarding the ownership and inheritance of the land when Robert David Lion Gardiner quarreled with his sister and her daughter, Alexandra Gardiner Goelet, believing they secretly wanted to sell the island for development after his death. After Goelet and her husband began paying the large upkeep and tax costs of the island (estimated at $2 million per year), Mr. Gardiner refused to pay half, hoping that the government would acquire the land and make it an historic site. His relatives took him to court, in 1980 he was barred from visiting the island. On appeal, this was reversed in 1992, and Mr. Gardiner regularly visited the island, though he did not speak to the Goelets.
Mr. Gardiner, who claimed the title "16th Lord of the Manor of Gardiner's Island" and lived in East Hampton, married in 1961 but had no children, leaving him with no heir. Upon his death in 2004, his ownership of the island passed to his niece. The Goelets offered to place a conservation easement on the island in exchange for a promise from the town of East Hampton to not upzone the island, change its assessment, or attempt to acquire it by condemnation. If accepted, the easement would ensure that the island is not developed for the foreseeable future.
The island has the largest stand of White Oak in the American Northeast, as well as the oldest surviving wood-frame structures in New York state, a carpenter's shed built there in 1639. The island is home to New York state's largest colony of ospreys, and is one of the few locations in the world where they build their nests on the ground, as there are no natural predators to the osprey on the island.