Metallic nanoparticles show extraordinary strong light absorption near their plasmon resonance, orders of magnitude larger compared to nonmetallic nanoparticles. This “antenna” effect has recently been exploited to transfer electrons into empty states of an attached material, for example to create electric currents in photovoltaic devices or to induce chemical reactions. It is generally assumed that plasmons decay into hot electrons, which then transfer to the attached material. Ultrafast electron–electron scattering reduces the lifetime of hot electrons drastically in metals and therefore strongly limits the efficiency of plasmon induced hot electron transfer. However, recent work has revived the concept of plasmons decaying directly into an interfacial charge transfer state, thus avoiding the intermediate creation of hot electrons. This direct decay mechanism has mostly been neglected, and has been termed chemical interface damping (CID). CID manifests itself as an additional damping contribution to the homogeneous plasmon line width. In this study, we investigate the size dependence of CID by following the plasmon line width of gold nanorods during the adsorption process of thiols on the gold surface with single particle spectroscopy. We show that CID scales inversely with the effective path length of electrons, i.e., the average distance of electrons to the surface. Moreover, we compare the contribution of CID to other competing plasmon decay channels and predict that CID becomes the dominating plasmon energy decay mechanism for very small gold nanorods.