Tonight I've come across Virtutech, a company doing emulation of various hardware platforms. Using their words, they do virtualized software development.
Their products had me asking myself a question: can we really trust virtualized environments as being significant for our tests?
Last week I had a discussion with a colleague about building an exploit-testing machine where we should run new exploits, a simple sandbox for our lab. My colleague was arguing that using a virtualized solution could have a significant impact on tests involving direct access to memory at ring 0. I've not been able to find an answer to this argument (feel free to comment if you did) since technical insights on these details are somewhat lacking.
Model checking is a difficult discipline, seldom used in real world. Virtutech solution seems to be based on SimICS, a virtualization platform originally from SICS. SimICS is around since 1995 as a full platform emulation aimed at virtualizing embedded systems, and as such seems to be a rather reliable solution: inside its framework hardware vendors have to develope an emulation layer representing their hardware (a virtual platform).
One could ask how much reliable the framework is, and how much reliable the virtual platforms actually are. From Virtutech website:
It is important to note that a Simics Virtual Platform is a representation of the physical board/system. Virtutech does not warrant that all aspects of the physical hardware have been modeled. Consult the documentation accompanying the Virtual Platform for additional details regarding actual implementation.
That is: you cannot trust the platforms, and we're speaking about rather simple environments when compared to full x86 server systems.
So, the question is: can we really trust virtualization from a formal, rigorous viewpoint?
Would you trust a life-support machine tested only on virtualized hardware, to cut time to market?