The increasing demand of modern society for clean energy has made it critical to develop novel battery materials with high energy density and cycling stability. Cathode materials of Li-ion batteries have been extensively studied in the past, and recently it has been proposed that by utilizing the anionic redox reaction one may be able to increase the total capacity of cathode materials. But the mechanism behind this proposal is not well understood because the traditional point of view is that the redox reaction is mainly carried out by the transition metal (TM) ions and experimentally achieved energy capacity is much lower than what is predicted by theory assuming all the Li ions can be extracted from the cathodes. In this work, we carefully study the general charge-compensation redox process of cathode materials upon delithiation, based on the molecular-orbital analysis and first-principles calculations. We reveal that the charge compensation in a cathode during the delithiation-lithiation process is mainly achieved by the oxygen redox reaction, and only partially by the TM redox reaction, due to a negative-feedback effect and the associated charge transfer from the O 2p to TM d orbitals. More importantly, our band structure and total energy calculations indicate that the energy density in a cathode is self-limited by the structural stability of its host upon delithiation, which depends on the energy level of the highest occupied states (HOS) of the host. When a critical amount of Li ion and electrons are removed from the host, the HOS becomes low enough to promote the formation of oxygen vacancy and thus destroy the host structure. As a result, suitable TM elements have to been used in the cathode materials to adjust the HOS level to keep a balance between energy density and stability, because the HOS is mainly determined by the TM for a given anionic framework. This new physical understanding and insights provided in this work on the redox process in the Li-ion batteries is important for the future development of new cathode materials.