Ultrafast Spectroscopy of Single Plasmonic Nanostructures