In order to protect the security of the execution environment of security-sensitive programs in computing devices, researchers have proposed the trusted execution environment (TEE) technology, which provides security-sensitive programs with a secure execution environment isolated from the rich computing environment by hardware and software isolations. Side-channel attacks have evolved from traditional attacks requiring expensive equipment to now attacks using software to infer confidential information from its access mode obtained through microarchitecture states. The TEE architecture only provides an isolation mechanism and cannot resist this kind of emerging software side-channel attacks. This study thoroughly investigates the software side-channel attacks and corresponding defense mechanisms of three TEE architectures: ARM TrustZone, Intel SGX, and AMD SEV, and discusses the development trends of the attacks and defense mechanisms. First, this study introduces the basic principles of ARM TrustZone, Intel SGX, and AMD SEV, and then elaborates on the definition of software side-channel attacks and the classification, methods, and steps of cache side-channel attacks. Second, from the perspective of processor instruction execution, a TEE attack surface classification method is proposed to classify TEE software side-channel attacks, and the attacks combining software side-channel attacks and other attacks are explained. Third, the threat model of TEE software side-channel attacks is discussed in detail. Finally, the industry’s defense mechanisms against TEE software side-channel attacks are comprehensively summarized, and some future research trends of TEE software side-channel attacks are discussed from two aspects: attack and defense.