The 17th century garden was swept away in the mid 18th century and was replaced by a 'picturesque' design. This was maintained throughout the 19th century until the estate was sold in the 1960s. Since then the design of the gardens and grounds have deteriorated.
In the 13th century, the Manor of Yester was granted to an Englishman, Hugh Gifford. The property remained in the hands of his descendants until the mid 20th century. In the late 13th century Yester Castle was built by Hugo de Gifford about a mile to the south-east of the house. He is also said to have constructed the subterranean Goblin Ha' (Goblin's Hall) by magic. St Bathan's Chapel near the house was the parish church for the adjoining village of Yester. The village was moved to Gifford probably in the 17th century.
The Gifford heiress married a member of the Hay family and, throughout the 15th and 16th centuries, the Hay family were at the forefront of Scottish affairs. In the mid- 17th century the 1st Earl of Tweeddale considered building a new house at Yester. The 1st Earl fought for the Royalists during the Civil War and was created Earl in 1646; later he changed sides and served as a Scottish representative in two Commonwealth Parliaments. Following the Restoration and under the Duke of Lauderdale, the 2nd Earl was appointed a privy councillor and commissioner of the Treasury. He supported William III and, as a reward in 1692, was appointed Lord High Chancellor of Scotland. Two years later he was created 1st Marquess of Tweeddale and died in 1697. His son married Lauderdale's only daughter Margaret.
From the 1660s, the 2nd Earl began planting at Yester. He was a close friend of John Evelyn. Daniel Defoe wrote, in about 1727, that from 1664 over 6,000 acres were planted. In 1670 Sir William Bruce was consulted on a new house by the 1st Marquess but it was his son, the 2nd Marquess, who in 1699 instructed James Smith to start work on the central block of the new house. Building continued until about 1728. However by 1676, the park had already been enclosed and by the 1690s, before the new house was built, one of the most ornate formal gardens and one of the largest parks in Scotland had been laid out around the old house. They can be seen on several delightful oil paintings by the Dutch artist, De Witt.
The 2nd Marquess was also a politician and was, for a few months, Lord High Chancellor under Queen Anne. He was a strong advocate of the Act of Union. He died in 1713. His son, the 3rd Marquess, died two years later before the building work had been completed.
John, the 4th Marquess, inherited as a minor. He became involved in politics and was eventually appointed Secretary of State for Scotland . In 1729 building resumed and William Adam was asked to add a pedimented attic to the south front and an Ionic centrepiece to the north front (later replaced by Robert Adam). Under the Marquess' supervision John and Robert Adam worked on the house throughout the latter part of the 18th century. By the 1760s, the magnificent formal gardens had been swept away and John Adam was proposing 'Chinese Bridges and Temples' on the lawns.
In 1824 Loudon described Yester in his 'Encyclopaedia of Gardening' as 'an elegant and magnificent structure of stone, with a park containing some very fine old trees and a good kitchen garden'.
George, the 8th Marquess, followed Wellington throughout the Peninsular War, fought at Waterloo and rose to the rank of Field Marshal. He was also a recognised agricultural reformer. In the 1830s the front of the house was reorientated to the west side, the double pair of curving steps removed and the glass port cochere built by the architects Robert Brown and Son.
At the end of the century George's grandson William, the 11th Marquess, modernised the house with the assistance of R. Rowand Anderson. By the 1920s, the grounds were described by Thomas Hannan as: 'beautifully laid out with flower beds, tennis court and numerous stone vases on pedestals with a fountain playing near the tennis court'. Photographs show the details of the colourful borders. David, the 12th Marquess died in 1967 and shortly afterwards the estate was sold.