By way of background, one first needs to understand there is a difference of opinion between Ashkenazim and Sefaradim regarding the absorption and kashering of glass utensils.
Some poskim believe glass is hard and smooth (nonporous) therefore cannot absorb taste and doesn't require kashering. Others believe glass is similar to metals and absorbs taste, but maybe be kashered through hagalah (hot water). Others believe that since glass is made from sand, it is comparable to earthenware and cannot be kashered.
Ashkenazi poskim follow the strictest opinion for Pessah but many are lenient and permit kashering glass (if for instance meat fell into a hot milk glass). Sefaradim follow the first opinion that glass never needs kashering even for Pesach.
Once this is understood, there are two opinions regarding using one set of glass for meat and dairy
- Some use one set of drinking glass for both meat and dairy meals (additional leniency: glass are not used for hot meat drinks, they are merely placed on the table) (Yabia Omer YD 4:5) - what I heard from my rav is that one needs to be careful when putting dirty milk glasses in the sink not to pour hot meat soup on them
- Some Ashkenazim follow the strict opinion and have two set of glasses, now that glasses are cheap (Teshuvos v'Hanhagos 1:432, Tzitz Eliezer 9:26)
Everyone should ask his own rabbi how to proceed based on his personal circumstances.
Source: R Binyamin Forst's the kosher kitchen and here for R Chaim Pinchas Scheinberg's views.