In small saucepan over medium heat, add milk, lavender, monk fruit, and vanilla. Slowly bring to a gentle boil, about 15 minutes. It’s important to allow time for the lavender to infuse the milk. Once at a low boil, whisk in agar agar. Unlike gelatin, agar agar needs to reach 165 F to activate. Pour infused milk over fine mesh sieve, then carefully divide milk into 4 glass jars or small ramekins. Let slightly cool for a few minutes, then place in fridge to set, 40 to 60 minutes.