I'd be more concerned about soil contamination from lead, mercury, cadmium, arsenic and other heavy metals and petroleum products, from New Haven's industrial past, lead paint from houses, and asbestos from siding. If anything, your neighbor's waste water may be contaminating the groundwater, but it's probably not migrating to your garden. The phosphate, in their laundry water, is a fertilizer, though that's not necessarily a good thing.
In addition, that house is a dump and the owner has multiple foreclosures in the neighborhood.