In a mixing bowl, mix together vanilla pudding, milk, and sweetened condensed milk. Let the mixture rest for 10 minutes.
Mix in sour cream and Cool Whip. If using, mix in yellow food coloring until thoroughly incorporated.
Beat cream and powdered sugar together into stiff peaks (this will help it stand up in the trifle). This is easiest with a stand mixer, but you can do it with an egg beater or hand mixer as well.
Assemble your trifle in this order: an even layer of Nilla wafers, an even layer of fresh banana slices, 1/3 of the pudding, and 1/3 of the whipped cream. Repeat 2 more times, then top with the remaining Nilla wafers and banana slices.
Chill at least 2 hours, but you may also do overnight. Serve to a satisfied crowd.