In this paper, a numerical and mathematical model was performed to evaluate performance of photovoltaic thermal nanofluids system. The energy balance equations for every layer of the photovoltaic thermal system are solved using numerical simulations. The objective of this work is to study theoretically a new configuration of the photovoltaic thermal system which includes stainless steel rectangular tube and nanofluids as a working fluid for extracts heat from photovoltaic panel. Three different volume concentrations of Titanium Oxide, TiO2 from 0.5 to 1.5%v with effects of mass flow rate and solar irradiance were studied. Results indicated that the electrical and thermal efficiencies is proportional to mass flow rate and the best results achieved with lower volume concentration. Nanofluids provide a higher performance due to higher specific heat, Nusselt number and heat transfer coefficient. It is also found that, volume concentration of 0.5%v under average solar irradiance of 650W/m2 with equal to average climate in Malaysia has shown the higher efficiencies in electrical and thermal respectively.