4,4^,-Bismaleimidodiphenylmethane (BMIPM)/O,O^,-diallylbisphenol A (BA) system was modified by organic rectorite (OREC) to develop a novel BMI/BA/OREC nanocomposite. The effect of OREC on the viscosity and reactivity of BMIPM/BA system was investigated. The mechanical properties of BMIPM/BA/OREC composites such as the flexural and impact strength were evaluated. The morphology of cured BMIPM/BA/OREC systems was investigated by X-ray diffraction (XRD), transmission electron microscopy (TEM), and scanning electron microscopy (SEM). The hot water resistance of BMIPM/BA/OREC systems was discussed. The thermal property of BMIPM/BA/OREC systems was investigated using thermogravimetric analysis (TGA). The dynamic mechanical properties of BMIPM/BA/OREC systems were also measured. Results show that the addition of OREC has a significant influence on the reactivity of the BMIPM/BA system. Proper content of OREC can improve the flexural strength, impact strength, and hot water resistance of a BMIPM/BA system. The addition of OREC cannot decrease the thermal degradation temperature of cured BMIPM/BA system with a slight sacrifice of the glass transition temperature (Tg).