What I don't love is the wind.
The desert wind can be incredibly viscous. Forty, fifty and even sixty mile an hour winds arrive in the desert with an alarming frequency. If you camp in the desert for more than seven days, you will definitely experience a wind storm.
The wind seldom has an impact on climbing. You can usually find a crag that is sheltered. Camping, however, is another story.