Rather than analyzing lesions individually, clinicians frequently assess lesions to biopsy by considering them versus the rest of the lesions on the body. An odd lesion among many similar lesions is thought not to be as dangerous as an odd lesion on a patient whose other lesions are more benign. This is known in dermatology as the ugly duckling sign and it is frequently used to diagnose melanoma, especially in patients with many atypical nevi. Until now, it has not been possible to explore the ugly duckling concept with machine learning due to the lack of large datasets with multiple labeled images per patient. We present the first dataset of melanoma and comparative benign lesions within the same patient to support new machine learning challenges, and further pave the trail of artificial intelligence in the diagnosing of skin lesions, with the hope of someday deploying AI in clinics to assist dermatologists throughout the world. Images were collected from Memorial Sloan Kettering, Melanoma Institute Australia, University of Queensland, Medical University of Vienna, University of Athens, and University of Barcelona.