In a saucepan over medium heat, combine sugar, butter and milk. Bring to a rapid boil and boil for one full minute. Remove from heat and stir in the vanilla and peanut butter. Mix in the oats, stirring until the mixture begins to cool. Transfer to a large bowl if it does not fit into the pan well. Drop batter by teaspoonfuls onto waxed paper. Let cool until set.