Luminance is the most fundamental visual information. In comparison with other visual attributes, we know less about the neural mechanism responsible for coding luminance due to that visual neurons weakly respond to uniform luminance and many of them do not respond. Responses of a portion of neurons to luminance in the primary visual cortex are slower than those to contrast, which is thought to subserve the perceptual brightness induced by contrast at the borders of uniform luminance. However, our recent studies show that luminance responses of many neurons in the primary visual cortex are faster than contrast responses, and these neurons are more sensitive to lower spatial frequencies, higher temporal frequencies, and faster motion speed. The results indicate that the subcortical inputs of low spatial frequency and high motion speed channels contribute the generation of luminance responses of these neurons. It is known that responses to low spatial frequencies precede those to high spatial frequencies, which is thought to underlie the coarse-to-fine process of information processing in the early visual pathway. The early luminance response we reveal is the response to the extremely low spatial frequency, which is consistent with the temporal course of responses to low-to-high spatial frequencies. The rapid luminance response is the initial step of the coarse-to-fine process and the neural basis for processing the earliest coarse visual information. Moreover, the primary visual cortex contains a group of neurons that prefer luminance decrements and high motion speeds. The activities of these neurons may aid subjects to detect dark objects that are moving at high speeds under dim environment and benefit behaviors of subjects in the visual context.