How can a teenage girl, who recently started practicing the laws of tzniut as well as feasible, balance those laws with the prohibition against embarrassing her family members, who are embarrassed to be seen with her dressed so oddly?
To be tzanua in appearance leads one to look refined, with tidy hairstyle, upright posture, clean and proper attire that covers what needs to be covered according to halacha. Being well-dressed and well-mannered is also part of tzniut. How can that be referred as dressing oddly or be a source of embarassment? On the contrary, it will project to others that you should be taken seriously.
If we discuss tznius as a trait, as opposed to a minimum standard which must be met to avoid violating an issur, then I think your asking a good question to be successful. Part of the trait of tznius is trying not to make oneself conspicuous. Of course this isn't an absolute and cannot always be achieved, (head/hair coverings, traditional styles, hot days, etc.) but is still a good goal. Ideally (unless your part of a community with specific standards of dress that should be honored) your manner of dress should only be conspicuous when observed over a period of time.
I think that the following would be helpful in analyzing an outfit:
Make certain what needs to be covered is covered as needed during normal daily activities.
Select individual garments that wouldn't seem out of place.
Put them together in a way that wouldn't seem out of place on a moderate fall or early spring day.
Err on the side of less casual dress when practical. Part of the art of tznius is balancing taking care of your appearance without trying to attract others.
I understand that layering is pretty popular right now, and helps "kasher" otherwise unwearable garments, I suspect that you will have more success in "fitting in"/"not appearing odd" if you rely more on garments that are acceptable as designed.
Finding acceptable clothes can be a challenge, but I'm incline to believe that you can still find acceptable clothing that you can put into conventional outfits that would not stand out to a by-passer. Your friends and family will notice your habits but I suspect that they will have a much easier time accepting them.
Tznuit does not have to be "funny" looking clothes. When I was a teenager, I went through a modest dressing phase and actually eventually discovered a personally quirky style in it! While I'm not currently observing complete tznuit in dress...
1) Check out styles that might easily be modified for modesty. If you're more of an artsy, flowy type, you could try some of the neo-hippy type maxi dresses and sweaters, or funky long sleeved tees!
2) Another style that lends itself well to modest dressing is vintage. Skirt lengths are longer, often you can find lovely tops with flattering decoration that is more at the neckline or the wrists instead of plastered across the chest.
3) My personal fave (and the one I still exhibit when I can be bothered to "dress up") is a more preppy style. Oxford shirts, cardigans, plaid skirts (they do come in lengths other than "short" although you may have to look around a little).
And remember that while your personal sense of tznuit may mean "don't dress flashily" it doesn't have to mean "dress frumpily." You can and should feel comfortable in your clothes, and wear clothes that are generally flattering (which doesn't have to mean skimpy, or alluring).
You are to be commended for taking on a socially-challenging mitzvah. It's not always easy to be Jewish and be seen as different, whether it's through dress, food, or how you spend your Friday nights and Saturdays.
With any observance that sets you apart from others, take care in how you talk about it. It's about you, not about them, especially for your secular friends. Make it clear that you're doing this for your own reasons; you need to steer clear of anything that will be perceived as a superiority attitude, because people don't like to be around people they think are judging them. Since that is not your attitude, this just means listening to how you talk and imagining how it will be perceived. One simple, truthful explanation you can offer is "I find these clothes more comfortable".
(Some of your friends are probably Jews who you might wish were also keeping tziniut, so perhaps there is some subconscious judgement there. Don't ignore them, but you need to get comfortable yourself before you can be effective at drawing others in. Being a role model will likely work better than trying to persuade them.)
Another thing you can do is to spend time with your female friends (without any guys around); in those settings the rules are more relaxed, so a swimming suit, for example, is not a problem. That won't work on beaches and at public pools, but it could work at a spa, for example.
Finally, if you act like this is (1) unremarkable and (2) not open for debate, eventually people will lose interest in arguing with you. If you don't make a big deal about it then, in time, they won't either.