Unless you're an idiot who sprays oil all over the bearing seat you don't have to worry about oil damaging plastic. The oil goes inside the bearing not all over the bearing. As long as you only put the oil inside the bearing it's not going to ever touch the plastic.