Combine olive oil, lemon juice, garlic and rosemary in a small bowl. Season the lamb with salt and pepper and cover with marinade. Marinate at least 1 hour, overnight if possible. Discard the marinade, then grill over medium-high heat to desired liking, about 5 minutes on each side or broil in the oven.