A combination of their decent kiting capabilities and usage of Spell Breaker makes them relatively difficult to take down.
They are actually decent monks forming a "circle of protection" with the skeleton warriors. They cast Life Bond/Live Vicariously and heal with Infuse Health on the warriors and the warriors attacks heal them back up to full. Very little casting is needed.
Killing the Skeletal Berserkers first makes pulling the monks down very easy since they don't have any direct healing skills.