Good settings for highest HD video quality in VLC?

[Deathbliss]

I know this question is kind of general so I will elaborate.

I have a 720p video. I play it in VLC on my computer (latest build.) I have an AMD 7800 series video card. Running Windows 7 64-bit on an Intel quad-core processor. Performance is not an issue. Quality shouldn't be either, running at 1080p through HDMI hooked directly to my 32" LED LCD TV.

When I play this video and it zooms in to something like someone's face, I see lots of blocks and squares. There's some sort of strange smoothing effect going in - it's very bad up close.

Now I take this exact same video and play it in 1080p through HDMI on my KDLinks HD 680 and the skin looks almost perfectly smooth up close. I see some faint evidence of little squares but there is a major difference in video quality compared to VLC.

I know some upscaling is going on. I followed this guide at YouTube so I could play DVDs without too much problem:
http://www.youtube.com/watch?v=t7AcAzHdPEo

I have audio and video set to DirectX and primary devices. Latest video card drivers installed. No other AV settings touched in VLC. I want to know how to make the picture in VLC as nice as my KD Links HD680. I do not want to stream or hook the media player up or anything. I want to understand what settings I need to tweak in VLC so that HD video plays correctly, because as of right now it is not.

Appreciate the help!

[mindbomb]

since you are on windows and seem to really want the best, may I suggest mpc hc with madvr?
http://www.avsforum.com/t/1357375/advanced-mpc-hc-setup-guide

(shameless plug to my guide)

[SamKook]

I'd be curious to know if VLC actually has settings to increased quality since I've always seen it as an install-and-never-have-to-touch-the-settings kind of player.

mpc hc(or mpc be which now has native wasapi support) with madvr would be the best thing you can get for quality though as mindbomb said.

[antonygrakham]

I'm not sure if this problem has a decision.