The ungrouped folder within in your contact groups is a house for those contacts who haven't been allocated to a contact group. These contacts can be sent to when completing a mass mailing (to all contacts) but not from the group 'ungrouped'.
You can remove these contacts from your account by selecting the group, and either clicking the tick box at the top of the screen to select all contacts, or clicking the individual tick boxes next to each contact if you only want to remove specific contacts. Once you've made your selection click on the Actions drop down box at the top and select 'Delete from Your Account'
Removing these contacts from your account will not unsubscribe these addresses, so if you decide that you have a home for these contacts they can be re added to your account.