I was recently told by an old New Englander that her grandmother used to come home from church and stay in her "Sunday Best" all day, even while tending to her garden. Keeping Sunday holy is something we don't practice anymore as a generation. Yet, there are those still out there that keep this day sacred.
So go on, wear your dress all day. Do things a little bit fancy. Keep Sunday special from all the rest.