Sex differences in the brain have long been a hot topic in neuroscience. Previous studies on gender differences have focused on brain structures and functions related to higher-level cognition, but less attention has been paid to low-level sensory and perceptual systems. Recently, a growing number of studies have shown that men and women also show remarkable differences in visual processing. In this paper, we started by reviewing the behavioral and neural evidences supporting the notion of sex differences in the visual system. We then proposed two potential explanations and discussed the evolutionary benefits of sex differences in visual perception. Finally, we highlighted the importance of incorporating sex as a biological variable in visual neuroscience research, and offered a few suggestions for future studies on sex differences in the visual system.