Preheat oven to 350.
Lightly grease a 24-mini muffin pan with baking spray and set aside.
Combine flour, sugar, baking soda, baking powder and salt in a mixing bowl; whisk until well combined. Set aside.
In a separate mixing bowl, combine eggs, milk and vanilla; whisk until thoroughly combined.
Add melted butter and continue to whisk until well incorporated. Make sure your butter isn't HOT!
Gently fold the egg-mixture into the flour-mixture. DO NOT overmix. If your muffins are dense, it is because you overmixed. Mix just until everything is wet and combined.
Spoon batter into the previously prepared muffin pan, filling each cavity about 3/4 full.
Bake for 10 to 11 minutes, or until a toothpick inserted in the center comes out clean.
Remove from oven and let stand 2 minutes in pan.
Transfer to a wire rack and let slightly cool.
In the meantime, prepare the glaze.
Fill a bowl with hot water and set aside.
Place milk, vanilla and powdered sugar in a small saucepan; cook over medium heat, constantly stirring just until all is well combined and no lumps appear.
Remove from heat and set saucepan over the bowl filled with hot water.
Ice the muffins with the glaze, one at a time and transfer back to cooling rack. Place a piece of foil or paper towel underneath the cooling rack because some of that glaze will drip off.
Serve.