Thermal management of lithium-ion batteries is crucial for safe and reliable performance. Quantification of thermal transport in lithium-ion batteries is challenging due to the heterogeneous and multi-scale nature of the constituent materials. The electrodes themselves typically consist of particles with micro- and nano-scale features, and the structure and composition of the electrodes evolve with charging and discharging, and degrade with repeated cycling. Integrating thermal sensors within batteries is challenging due to the confined geometry and possibility of shorting the electrodes if using thermocouples within the closely packed electrode stacks. Here, we focus on quantifying thermal transport in batteries using infrared microscopy through the design of battery cells with infrared transparent windows for viewing the internal thermal distributions. In parallel, we measure the thermal conductivity of the battery cells in an open cell configuration without charging and discharging. These new data provide insight into the performance, heat generation, and thermal transport within the battery cells. Ultimately, these properties are required for understanding thermally driven failure modes, for designing systems with improved safety and reliability, and to prevent thermal runaway.