Endermen are very difficult to kill I've found even with a diamond enchanted sword. So to save the levels and swords, I've resorted to using creepers. Get an Enderman inbetween you and a creeper (keep cursor on the ground to avoid the Enderman attacking you) and have the creeper blow up. I've found it takes 2-4 creepers to kill an Enderman depending on how close you get them to explode. And since its the environment that is technically attacking, the Enderman will no react.
Good luck. I've killed 7 Enderman this way successfully and gotten 5 pearls. TnT also works really well too.