Lateral geniculate nucleus
Anatomy   Wikipedia  
A simple mnemonic for remembering this is "See I? I see, I see," with "see" representing the C in "contralateral," and "I" representing the I in "ipsilateral." Another is "Emily and Pete meet eye to eye" as in "M and P meet I to I," or again, Magno and Parvo meet Ipsi to Ipsi.

Another way of remembering this is 2+3=5, which is correct, so ipsilateral side, and 1+4 doesn't equal 6, so contralateral.