The castle was probably built by Sir Simon Fraser of Oliver Castle (East Lothian) beteen 1263 and 1266 while he held the office of High Sheriff of Tweeddale. The castle was acquired by the Hays of Yester who rebuilt the structure c. 1370. It was held by them until when it was bought by William Douglas, 1st Duke of Queensberry from the first John Hay 2nd Earl of Tweeddale (later first Marquess of the same). It was visited by Mary, Queen of Scots in 1563, and by her son James VI in 1587. The castle was attacked by Oliver Cromwell, and required the longest assault on any stronghold south of the river Forth to force it to surrender. By then cannon fire had damaged it severely. Major changes to the castle came about during the restoration by the Marquis of Tweeddale. He was a great agriculturalist who planted a fine avenue of yews, of which one side remains. On the death of the fourth Duke of Queensberry the castle was inherited by the Earl of Wemyss and March (although the dukedom went to the Scotts of Buccleuch) Neidpath still belongs to the Wemyss family; the second son of the family takes his courtesy title from it. William Wordsworth and Sir Walter Scott both visited the castle in 1803.
The castle is still owned privately by the Earl of Wemyss and March.