In the late 1700s national galleries were all the rage in Europe. A number of countries nationalised their royal collections but the British government instead wanted to purchase a major collection when one became available. From 1777 onwards they missed a number of opportunities but finally bought the 38 paintings in the collection of the late John Julius Angerstein (1735–1823). This was displayed in his former townhouse at 100 Pall Mall, opening to the public on 10 May 1824. The house is lost but Wikipedia has a drawing which shows it to have been a not particularly large, traditional, Georgian terraced house.
The house was too small to accommodate all the visitors and in 1832 construction began on a new building by William Wilkins, a short distance away, overlooking the newly constructed Trafalgar Square.
Someone was looking to save expense wherever possible: the portico came from nearby Carlton House, demolished in 1826; when the first duke of Chandos, had to sell off his Canons estate in Little Stanmore it was demolished in 1747 and its colonnade was used on the front of the Gallery; cost-savings on the Marble Arch meant there were spare sculptural elements, some of which were put to good use on the front elevation.
The Carlton House columns were used for the two side porticos which gave access to the workhouse and a barracks immediately behind (the site allowing the gallery to be only one room deep). The eastern half of the building housed the Royal Academy until 1868. Since first construction the gallery has extended northwards and westwards as can be seen in this floor plan from Wikipedia, author Gringer.
Since opening with 38 paintings the collection has been expanded by donations and purchases and now holds more than 2,300 paintings dating from the mid-13th century to 1900.
Charles Eastlake was a key figure in the early management of the gallery and expansion of its collection, and in 1855 was appointed its first Director, a post he held until his death, 10 years later.
2024: Richard Rogers' design of the Sainsbury Wing (to the west) was very controversial, with Prince Charles publicly describing it as a "monstrous carbuncle on the face of a much-loved and elegant friend". It was funded by the Sainsbury brothers. During construction Lord John Sainsbury also took against the design, particularly two unnecessary columns flanking the foyer. His brother Simon convinced him to go ahead with the funding but on condition that John was allowed to place a note inside the columns stating his objection. Which he did, in 1990, and recently, with the Wing undergoing renovation works, those columns have been demolished and this note found. NPR have the full story and the text of the note.