The cadenza was originally a vocal flourish improvised by a performer to elaborate a cadence in an aria. It was later used in instrumental music, and soon became a standard part of the concerto. Originally, it was improvised in this context as well, but during the 19th century, composers began to write cadenzas out in full. Third parties also wrote cadenzas for works in which it was intended by the composer to be improvised, so the soloist could have a well formed solo that they could practice in advance. Some of these have become so widely played that they are effectively part of the standard repertoire, as is the case with Joseph Joachim's cadenza for Johannes Brahms' Violin Concerto and Beethoven's set of cadenzas for Mozart's Piano Concerto no. 20.
Concertos are not the only pieces that feature cadenzas; Scena di Canta Gitano, the fourth movement of Nikolai Rimsky-Korsakov's Capriccio espagnol, contains cadenzas for violin, harp, clarinet, and flute in its beginning section.
Nowadays, very few performers improvise their cadenzas, and very few composers have written concertos within the last hundred years that include the possibility of an improvised cadenza.
A famous example of a cadenza is Rachmaninov's Third Piano Concerto, in which the first movement features a long and difficult cadenza.
Another famous cadenza appears at the end of the first movement of Grieg's Piano Concerto in A minor. It is a long and impassioned cadenza which ends with the orchestra and piano playing together in a dramatic and rousing finale.
Cadenzas are also found in instrumental solos with piano or other accompaniment, where they are placed near the beginning or near the end or sometimes in both places. (e.g. "The Maid of the Mist," cornet solo by Herbert L. Clarke, or a more modern example: the end of "Think of Me" where Christine Daač sings a short but involved cadenza, in Andrew Lloyd Webber's The Phantom of the Opera.)