The Nezu Museum (which is near Aoyama, not Nezu) art collection is mainly Asian artifacts, displayed in a few small galleries. While certainly pretty, they are no match for the extraordinary gardens.
The Japanese gardens are filled with momiji, bright green in the spring and summer, and shockingly red in the autumn. The trees conceal a wealth of interesting statues and traditional teahouses (including one in a small boat). The whole ensemble is simply breathtaking.
The gardens can be enjoyed year round, even when it snows. The museum also has a lovely cafe, which overlooks the gardens.
Nezu Museum: http://www.nezu-muse.or.jp/en/
Google Maps: https://maps.app.goo.gl/weB7GGMd89QQMXfXA
Want to find out about more interesting places in Tokyo and Yokohama? Check this out: http://moveovergodzilla.blogspot.jp/