A surface plasmon in a metal nanoparticle is the coherent oscillation of the conduction band electrons leading to both absorption and scattering as well as strong local electromagnetic fields. These fundamental properties have been exploited in many different ways, including surface enhanced spectroscopy and sensing, photothermal cancer therapy, and color display generation. Chemical synthesis and assembly of nanostructures are able to tailor plasmonic properties that are, however, typically broadened by ensemble averaging. Single particle spectroscopy together with correlated imaging is capable of removing heterogeneity in size, shape, and assembly geometry and furthermore allows one to separate absorption and scattering contributions. In this talk I will discuss our recent work on distinguishing the different contributions that cause plasmon decay, and especially plasmon damping due to energy and charge transfer from the metal to molecular, semiconducting, and metallic acceptors.