From a letter to a friend, 1981:

...The whole
notion of attractiveness is one of nature's most transparent cheats.
Sensible people decide whether to get close to each other according to
the pleasure they can reasonably hope to give each other when they are
close, _not_ according to the pleasure they happen to give each other
at a distance. The two are very poorly correlated. (Note in this
connection that _attractive_ & _repulsive_ are not good opposites. It
is a good bet that anything unpleasant will be less so at a greater
distance. It is a bad bet that anything pleasant will be more so at a
smaller distance.) Cheat back! Look at the rose, but eat the apple.

...

There are two special causes for hopefulness in applying these ideas
to sexuality. One is that tho not everyone can be a rose, almost
everyone can be an apple: the things that matter most after you have
taken your glasses off are more easily altered to suit the customer
than the more or less standardized displays. The other is that
nature, if cheated gallantly enough, is a good loser: those who have
given you sexual or other pleasure actually start looking better as a
result, even at a distance.

What is "sexual attraction" anyway, that we should respect it so
much at the cost of so much misery? Some of it (e.g. tits & ass, i
suspect) is wired in genetically & is almost wholly arbitrary -- once
it got started, it was selected for because it was selected for
because etc.; sex is like that. Some of it is mere social convention,
& some is a mixture (e.g., i imagine society determines what counts as
a signal of high status, but genetics makes high status as such
attractive). But all of it is remarkably plastic, and our usual
failure to mold it according to reason is only another reminder that
wherever sex is concerned we must expect concentrations, not only of
what is beautiful, but also of what is cruel, gratuitous, &
preposterous.