With the ability to deform in response to specific stimuli, polymeric hydrogel actuators are important bionic materials. However, because shape deformation is derived from the diffusion of water molecules, the response rate of hydrogel actuators is usually slow, which severely limits their potential applications. In this work, a thermoresponsive PNIPAm hydrogel sponge is developed, and the hydrogel sponge shows ultrafast de-swelling/swelling capacity (equilibrium time 7 s) and large deformation degree (40%) due to the assistance of capillary force. Hydrogel sponge actuator with rapid response performance is further constructed, and the hydrogel sponge actuator could be used to capture moving objects and living creatures. Moreover, light-controlled directional movement can be achieved by incorporating photothermal functional components into the hydrogel sponge actuator. This work would promote the application of hydrogel actuators in soft robots.