In a saucepan, bring the milk, vanilla bean (split and scraped) or vanilla extract, and lemon zest to a boil. Remove from heat and let infuse for 10 minutes.
In a medium bowl, whisk together the egg yolks and sugar until pale and creamy.
Add the cornstarch to the egg mixture and mix well.
Remove the vanilla bean and lemon zest from the milk. Gradually pour the milk into the egg mixture, whisking constantly.
Pour the mixture back into the saucepan and cook over medium-low heat, stirring constantly, until the cream thickens.
Transfer the pastry cream to a bowl, cover with plastic wrap (pressing it directly onto the surface to prevent a skin from forming), and let cool completely.