La Derniere Goutte is also great for the Loire and the Northern Rhone and is the center of a great expat 'scene', along with Fish.
If you have the time, this is your chance to further explore the natural wines as Paris is ground zero for the scene.
Here's a great website with an overview of some of the more high profile places. Lots of the hipper, trendier ones are concentrated in the east and the 11th and the 20th but they are also spreading across the city.
http://www.morethanorganic.com/natural-wine-in-parisCaves Auge is one of the oldest and most established shops in this vein. In fact, they are the people who advise Lavinia, which has a fine selection, like an expensive Wall Mart, and apparently has a great tasting bar and restaurant where you can drink any bottle from the shop without corkage. However, they area bit more expensive than other shops and also suffer from the Wall Mart feel and lack of detailed knowledge from the staff, if that's something you care about.
La Cremerie (not far from La Derniere Goutte) is one of the more established natural wine bars/shops with one of the more 'extreme' selections (i.e. very few things that are actually drinkable, in my opinion) but is a nice place and a great way to see some of the newest producers before they get on the international radar. For some reason they are not included in the above website, perhaps because the guy who runs that website and who owns Racines is a refugee/former employee of La Cremerie.
Also in that list, I recommend Cafe de la Nouvelle Mairie. It is in a great square behind the Pantheon and has great food and good drinkable natural wine in a very civilized atmosphere, lots of regulars, without the scruffiness of some of the trendier spots in the 11th and 20th. Which are also fine, if you want that atmosphere.
For more traditional shops I HIGHLY recommend Les Caprices de l'Instant in the Marais near the Bastille (12 Rue Jacques Cœur in the 4th), Rafael is one of the most interesting cavistes in Paris, one of his parents is from South America (but I forget which country) and if you speak French you could easily spend hours chatting about everything. More importantly, they make it their policy to hold wines from previous vintages so check their book for the Bordeaux, Burgundy, Loire, Rhone from all the 90s vintages at relatively fair prices.
Bacchus et Ariane in the Marche St Germain (across the street from La Cremerie) is another great place with a small but well-chosen selection. La Cave du Panthéon is also great, like La Derniere Goutte it is in-between 'classic' and 'trendy'..
Speaking of which, nobody has mentioned the real 'classic' shops like Legrand which never really interest me but might be worth your time. Depending on how many other things you have going on. And it is easy to be busy in Paris!