I worked in the nuclear field for quite a few years. The building and safe operation of fission reactors be they PWR, gas cooled or heavy water is not the problem if the designs are good and the people well trained.
The main problem is the waste. This is not just the actual uranium which can to a certain extent be re-cycled but also the material in contact with it. When a nuclear power plant is decomissioned all the primary pipework needs to be disposed of in a safe manner as it is highly irradiated.
Simple enough but there is far more of this material than actual spent fissile material and that produces the problem of volume and size. The pipework would need to be cut into manageable sizes for burial/disposal.