Did your neighbors actually say something to you about it? My buddy did his front yard in native grasses and gravel two summers ago and his next door lady honestly thought he was broke and trying to save money on water. He told her the whole thing cost him more than sod would have, and she just blinked at him. The funny part is now she keeps asking where he got his rocks because hers look bad next to his. People see dirt and assume wallet problems, but a real xeriscape with good stone and healthy plants is not the discount option, it is the upgrade nobody recognizes yet.