I should start out by saying that I am a big fan of GT3, but I only played GT2 a little bit and hardly played GT at all.
With that out of the way, I think GT4 is a fantastic game. To be fair though it is only really the culmination of what GT3 could have been and isn't a huge leap forward. That is ok though, GT3 was released 4 years ago and probably took a couple of years at least to write, so a lot will have been learned since then about exploiting the PS2 to the fullest. The addition of a huge library of cars and a good number of tracks only improves the experience. Disappointingly absent is online play, instead we get LAN play and a particularly silly photo mode.
Very few games have equalled even the performance of GT3, but GT4 goes better in looks, sounds and feel and will certainly be tough to beat.
I happen to have a Logitech GT Force wheel, which was the official GT3 wheel and it makes any racing game many times more fun than with a controller, but the Gran Turismo games get a particular boost because they are so technical. Snow and rally tracks are something of a challenge to whip the wheel round for opposite lock, but that tradeoff buys you exceptional control for drifting round the corners and seems to be well worth it. I just got a gold on the IA license snow track after getting a bronze first time and then two silvers, so the wheel must definitely be helping because I'm not normally that good ;)
Therefore, if you are looking for a really good, but quite serious driving game, look no further. If you enjoyed GT3 a lot, this would be a good time to refresh the memory with a slightly new twist :)
In addendum, it is said that engineering samples of the PS3 have been produced, so let's hope that one is on its way to Polyphony Digital so they can blow us away with a stunning evolution in the GT series to launch with the PS3 (including full and amazing online support damnit!)
GT4 - IA license - snow track (Gold).
Click for a larger version