Story, first published: Galaxy Science Fiction, Feb. 1960
"Crying Jag" ... is the outrageous story of Wilbur, an alien who gets drunk on sorrow. The janitor at the local sanitarium decides that Wilbur would make the perfect confessor and cons the local psychiatrist into paying $7000 for Wilbur's services. Another alien, Jake, warns that Wilbur is a renegade on his home planet, but the janitor ignores Jake's warning, telling Jake that he will take all the aliens he can supply. The next morning the Earth is overrun with alien confessors - everyone on Earth can have an alien of his own.
Ewald, Robert J.: When the Fires Burn High and the Wind is from the North, p.56