Step 1

Place a rack in middle of oven; preheat to 350°. Lightly butter a 9"-diameter cake pan and dust with all-purpose flour; tap out excess. Line bottom of pan with a parchment paper round. Arrange 3 medium red plums, cut into ½"-thick wedges (about 8 oz.), in a circular pattern in pan, starting from center and working outward and overlapping if needed.

Step 2

Whisk 1 tsp. baking powder, 1½ cups (188 g) all-purpose flour, and 1 tsp. Diamond Crystal or ½ tsp. Morton kosher salt in a large bowl. Combine 2 Tbsp. finely grated lime zest and 1½ cups (300 g) granulated sugar in a large bowl and rub together with your fingers until evenly distributed, sugar starts to clump, and mixture is very fragrant. Add ¾ cup (1½ sticks) unsalted butter, room temperature, and beat with an electric mixer on medium-high until light and fluffy, about 3 minutes. Add 2 large eggs, room temperature, one at a time, beating to combine between each addition, then add 1 tsp. vanilla extract and beat 10 seconds. Reduce speed to low and gradually beat in dry ingredients, scraping down sides of bowl as needed. Add ½ cup whole milk, room temperature, and mix just until combined. Set batter aside.

Step 3

Bring remaining ½ cup (100 g) granulated sugar, remaining 1 tsp. Diamond Crystal or ½ tsp. Morton kosher salt, and 3 Tbsp. water to a boil in a small saucepan over medium heat. Carefully add ¼ cup fresh lime juice, watching out for spatter, and cook undisturbed until liquid is thickened and beginning to turn a deep amber color, 5–7 minutes. Gently stir, remove from heat, and immediately pour lime caramel evenly over plums in pan. Scrape reserved batter over and smooth surface if needed.

Step 4

Bake cake until golden and a tester inserted into the center comes out clean, about 1 hour. Transfer pan to a wire rack and let cake cool in pan.\

Step 5

Run an offset spatula along edges of cooled cake to loosen. Set a platter upside down over pan and flip over to release cake. Carefully remove pan and peel away parchment paper; discard. Serve warm or room temperature, topped with scoops of vanilla ice cream if desired.