Step 1

Preheat the oven to 350 degrees F (175 degrees C). Line a 12-cup muffin tin with paper liners.

Step 2

Cream sugar and butter for cupcake together in a medium bowl. Beat in eggs, one at a time, then stir in vanilla extract. Combine flour and baking powder; add to the creamed mixture and mix well. Stir in 1/2 cup milk until batter is smooth. Pour or spoon batter into the prepared pan.

Step 3

Bake in the preheated oven until a toothpick inserted into the center of each cupcake comes out clean, 20 to 25 minutes. Cool in the tin for 5 minutes. Transfer to a wire rack and let cool completely, about 20 minutes.

Step 4

While cupcakes cool, combine cold milk and pudding mix for topping in a bowl. Mix until smooth. Let pudding mixture sit for 5 minutes.

Step 5

Mix 1/2 cup whipped topping into pudding mixture. Pipe the pudding on top of the cooled cupcakes. Place a dollop of remaining whipped topping on top of each cupcake. Garnish with marshmallows and sift cocoa powder over tops.