The survey plan of 1764 by Peter May shows a formal design. The landscape was informalised prior to 1878, possibly by Robert Robinson, and designs were also drawn up by Thomas White in 1789-90. Many architects have also been involved at Cullen, including William Adam, Robert Adam, James Playfair and David Bryce.
The earliest records of a house at Cullen go back to 1232. Originally, it belonged to the St Clair family but later passed to the Ogilvies. In 1543 the Collegiate Church of Cullen was founded and its canons were provided with an apartment and garden on the site. Parish records show that in 1600 the Laird's House at Cullen was begun.
This laird, Walter Ogilvie, was created Lord Ogilvie in 1616 and his son, James, was created Earl of Findlater in 1638. The 3rd Earl, who succeeded in 1669, made some additions to the house but his plans were not nearly as ambitious as those of his son who was created 1st Viscount Seafield in 1698 before succeeding his father as 4th Earl of Findlater in 1711. His improvements were continued by the 5th Earl and appear on a survey plan by Peter May commissioned in 1764, the year of the 5th Earl's death. The 6th Earl held the title for only six years before his death but was a keen agricultural improver in his time. The 7th Earl of Findlater, 4th Earl of Seafield, inherited the estates in 1770 at the age of twenty. He lived abroad much of his life following his marriage in 1779 but nevertheless took a keen interest in all decisions concerning the estate. Robert Adam was commissioned to prepare a plan for a new house whilst James Playfair was asked to prepare alterations to the design of the existing house. Thomas White prepared plans for the policies in 1789-90 in which Adam's designs for the site of a new house, circular office block and stable-block were incorporated and, whilst these particular ideas were not taken up, their suggestion of resiting the village of Cullen from around the church to its present situation around the harbour was acted on, although not until some years later.
The Earldom of Findlater became extinct on the death of the 7th Earl in 1811 when his cousin, Sir Lewis Alexander Grant of Grant succeeded as 6th Earl of Seafield. He too was a great agricultural improver and was responsible for moving the village and enlarging the harbour. Both he and the 7th Earl of Seafield, who succeeded in 1853, were known as the greatest planters in the district during their time as lairds. The 7th Earl commissioned David Bryce to design extensive Victorian additions and alterations to the inside of the house. Some of the interior work was later removed by the 11th Earl of Seafield's daughter who inherited the estates in 1915. During her time the gardens were well maintained but, in the latter years of her life and after her death in 1969, the estate went into a period of decline. The appointment of a new factor in the early 1970s however reversed this trend; a series of changes made included the sale of the house and 17 acres of ground to Mr Kit Martin in 1983. Since then, he has divided the house and service buildings into separate dwellings which have been purchased by private owners who retain communal ownership of the woodland garden in the ravine below the house. Lord Seafield succeeded in 1969 and lives on the estate at Old Cullen.