Assume Your Neighbor is Your Equal: Inverse Design in Nanophotonics