This may well be the case, especially as you begin to experiment with service learning. Some faculty think they must hold separate sessions with their few service learners in order to not monopolize the time of the traditional learners in class. Instead, why not use the service learners as extra resources in class? Ask them how theories play out in the real world; how people affected by or working with issues view them, compared with the authors of textbooks and articles; how literature is based in real life (or not); how it is to communicate with people whose lives are so different from yours. If you want to communicate directly with your service learners, set up an e-mail conversation as described above. You probably wouldn't want to set aside special "reflection days" in a class where there are only a few people working in the community. Instead, try to weave the information from the community into the body of the course.
[Note: If you are dissatisfied with the number of students who choose service learning, next time check to make sure there is a perceived equality between assignments for service learners and traditional learners. Also, be encouraging and enthusiastic when presenting the service option. Tell students how you think they'll benefit from a community-based learning experience.]