The energy rebound effect is regarded as an obstacle of achieving the expected target of energy-saving policies, especially under a rapid urbanization background in developing counties, such as China. This has become a substantial drag of sustainable development in some cities. Shanghai is the economic center of China, and it is also a typical energy import-dependent mega-city. Investigating the evolution of Shanghai’s energy-saving performance and the energy rebound effect is significant for the implementation of energy-saving policies in other similar cities of China and other developing countries. Using the state space model with time-varying parameters and based on the IPAT identity and the Solow residual approach, this paper is the first study to present a specific estimation on Shanghai’s energy rebound effect caused by technological progress. The results show that, during the period of 1991–2016, the average energy rebound effect of overall economy and secondary industry in Shanghai was 93.96% and 73.10%, respectively, indicating a high partial rebound effect. Most of expected energy saving caused by improved energy efficiency is offset by extra energy consumption caused by technological progress. Regarding tertiary industry, the average rebound effect was 146.61%, indicating a backfire effect. However, the average energy rebound amount of tertiary industry is less than that of secondary industry. In particular, there is an increasingly negative impact of the rebound effect of tertiary industry on energy conservation in recent years, with the sector’s rapid expansion and corresponding increase in energy demand. Furthermore, we estimate the carbon rebound amount (i.e., carbon emissions caused by the energy rebound effect) and find that, on average, the energy rebound effect caused 13.1% and 0.41% increases in carbon emissions in Shanghai and China, respectively. Therefore, mitigating the energy rebound effect can significantly reduce carbon emissions. Due to the substantial impact of the rebound effect, technological progress and energy efficiency improvement should not be the only way to achieve energy-saving target, especially in energy import-dependent mega-cities like Shanghai. Some supporting policies should be implemented to ensure that the expected outcome of energy-saving effort can be realized as far as possible.