In a medium bowl, stir together almond milk and chia seeds. Add coconut flakes and slivered almonds and stir well until combined. Cover with plastic wrap and chill in refrigerator for at least 2 hours. Before serving, stir pudding again. Divide between bowls and top each with a 1 teaspoon of sugar if desired.