We performed a thorough and quantitative investigation of spectral diffusion at low temperature in carbon nanotubes wrapped in PFO and embedded in a polystyren matrix, a system of interest for applications for quantum light states generation. We show that an intrinsic line shape typical of exciton-acoustic phonon coupling can be traced back down to the µs time-scale, while spectral diffusion yields a feature-less spectral broadening of the zero-phonon line. By tuning physical parameters such as temperature and pump power we investigate the line-shape deformation at several integration time-scales. Finally, we discuss the implications of these optical properties as regards the efficiency of the static and dynamical coupling of carbon nanotube to optical micro-cavities for single photon generation.