A crack or hole at the back of your closet may be why ants enter your closet from there. In addition, if you have uncleaned garments stacked in your closet, they may attract ants due to their sugary or salty odor.
Another reason may be moisture in the area surrounding or inside the closet.