Speech delays can be part of Asperger's. So can motor skills issues, and ADHD-type symptoms. And, I understand, all sorts of other things.
The key factor, from what I understand now, is that you can have all sorts of developmental challenges, but unless you have the social skills issues, you cannot be Aspie.... therefore, enough pieces minus social skills problems would likely be where Pervasive Developmental Disorder (PDD)-not otherwise specified comes in.