User-interface automation using computer vision for BIOS validation

BIOS validation is performed on an operating system to test the functionalities of all features available. Keyboard and mouse are the main inputs for entering test cases commands to validate BIOS functionalities. Every test case will be validated based on the correctness of the visual output on a monitor. This method seems to be very time consuming and is prone to human errors. Many validation engineers have to involved in performing the BIOS validation hence increasing the overall BIOS validation operating cost. The aim of this project is to design an independent automation system that is able to mimic human interaction in BIOS validation. The approach is divided into two main parts. The first part is the development of a keyboard and mouse emulator using Pro-Micro microcontroller. The emulator can be program to send user intended keyboard and mouse input to the operating system. The second part is image analyzer development. It is developed to mimic the human visual ability to compare the expected visual output with the tested visual output. OpenCV and OCR are used to develop the image analyzer. Finally, the (keyboard and mouse) emulator is combined with the image analyzer to automate the BIOS validation. The proposed system is fully automated and able to reduce the total test time by 87.5% with over 90% of OCR accuracy.