Scheele's Green and Paris Green were popular shades in the 19th century, and both were made from arsenic. Scheele's green was even used as a food colourant at one stage!
In damp conditions the arsenic can sublimate from its compounds in dyestuffs and can thus be absorbed by us.
Like lead, it's a cumulative poision, so the longer you live with your wallpaper, the sooner it will kill you!
More details here for the terminally curious.