There are two types of Contacts:
My Contacts
My contacts are contacts that you have explicitly created by clicking on the New Contact button. Contacts that you have edited from under the All Contacts section also appear here.
All Contacts
All Contacts are essentially all the contacts to whom you have sent emails or calendar invites. These also include all the contacts under My Contacts.