Elevation is a killer. Taos was at 7,000 feet, but I hadn't really noticed any tiredness while I was there. Of course, I didn't really do any demanding physical activity there.
Great Sand Dunes, at 8,100 feet, really took something out of me. Our campsite was about 1/4 mile from the dunes, separated by a broad, shallow, sandy creek that carried snowmelt down from the mountains. The kids played there before we tried to climb the dunes--eventually stopping about 1/3rd of the way up the first, 500-foot dune.