Previously, we have reported nanocomposite double network (ncDN) hydrogels by introducing bare or reactive inorganic nanospheres or nanorods into double network hydrogels. The obtained ncDN gels showed very high compression strength and toughness. However, the toughening mechanisms remains yet to explore. In this work, a comparative study is presented to provide detailed investigations on the polymer-nanoparticle interactions for ncDN gels with bare or vinyl-grafted nanoparticles. First, the effects of physical adsorption and/or chemical bonding of polymer chains to nanoparticles on the mechanical properties of the parent single network hydrogels of poly(2-acrylamido-2-methyl-propane sulfonic acid) (PAMPS) and polyacrylamide (PAAm) are compared. The nanoparticles showed significant toughening to PAAm gel than to PAMPS gel, due to the strong adsorption of PAAm to nanoparticles. Second, by using PAMPS-nanoparticle hydrogel as a host for in situ polymerization of AAm monomers, the obtained ncDN gels showed outstanding compression strength and toughness, with vinyl-grafted nanoparticles toughening more than bare nanoparticles. Detailed comparative analysis on the initial and ultimate modulus of the ncDN gels suggests that, after PAMPS network fracturing upon compression at high strains, the strong polymer-nanoparticle adsorption/bonding plays a critical role in the mechanical properties of the gels, with silica nanospheres working more effectively than ATP nanorods. TEM images revealed that, ATP nanorods were fractured upon large strain compression, while the silica nanospheres served as energy dissipation center. This study provides toughening mechanisms of nanospheres and nanorods for nanocomposite double network hydrogels.