Preheat the oven to 350 degrees F. In a large bowl, whisk together the flour, almond flour, brown sugar, baking powder, baking soda and salt. Set aside. In a 10-inch round cast iron skillet over medium heat, brown all the butter except for 1 tablespoon, stirring occasionally until the butter foam dies down and you can see the brown butter at the bottom of the skillet, about 5 minutes. Remove from heat and pour into a separate medium bowl. Add 2 teaspoons orange zest and 2 tablespoons orange juice to the butter. Allow the butter to cool, then whisk in the milk, eggs and almond extract. Whisk the brown butter mixture into the dry ingredients, stirring only until the batter is smooth. Grease the same cast iron skillet you used earlier with remaining 1 tablespoon butter, then pour the batter into the skillet. Top with half of the sliced plums. Sprinkle with raw sugar and the chopped almonds, and bake until the cake is a little firm, about 10 minutes. Remove the cake from the oven and top with the rest of the plums, pressing them down into the batter a bit. Put the skillet back in the oven and continue to bake until the cake is golden brown and a toothpick or skewer inserted into the center comes out clean, about 15-20 more minutes. Transfer the skillet to a cooling rack and cool for 15 minutes. Serve warm.