I think affirmative action was never intended to last indefinitely. It was an idea whose utility rested on its successfulness and whose successfulness, or lack thereof, may certainly redound back to its applicability.

Not that I necessarily agree with the way things are going, but I certainly think those affected negatively by affirmative action have recourse to the courts just like anyone else.