Music is an auditory art, which may play an important role in education and psychological development of children. Phonological awareness describes the ability to perceive, recognize, analyze, and manipulate phonological units, which is an important indicator to predict children’s literacy skills. This review summarized experimental studies on the effect of musical training on children’s phonological awareness in the last decade, and discussed the relevant neural basis and hypotheses. It has been demonstrated that musical training could not only promote children’s phonological awareness at the behavioral level, but also affect children’s phonological auditory processing ability at the pre-attention level. Music training may affect the neural basis of phonological processing from two aspects: on the one hand, it may affect children’s speech perception ability at the pre-attention level by strengthening the basic auditory neural pathway and the auditory cortex; on the other hand, it may affect the phonological coding and auditory-motor integration by enhancing the functional connectivity among regions related to phonological processing. The related neural mechanisms provide the biological basis for music training to promote children’s phonological awareness. Based on the existing researches and theoretical hypotheses, the review proposed a hierarchical model to explain the cognitive mechanism of the effect of music training on phonological awareness, which presents that there are three levels of the cognitive neural mechanisms of phonological awareness effected by musical training. The first level, musical training may promote the basic auditory processing of speech by strengthening the basic auditory neural pathways, in which training in rhythm may strengthen the perception of speech duration, and training in pitch may strengthen the perception of speech frequency. The second level, musical training may further promote speech coding by enhancing the neural networks for speech processing, in which training in rhythm promotes the recognition of consonant, rhyme, and syllable, training in pitch promotes the recognition of vowel, rhyme, and syllable. The third level, musical training may promote the speech motoric representation and auditory-motor integration by enhancing the interaction between auditory and motor cortex, further improve the quality of phonological representation. The potential theoretical significance of the effects of musical training on children’s phonological awareness was discussed in the context of revealing the cognitive neural mechanism and rules of human’s language development, and the potential application in the field of children’s education was also discussed.