Verificaton Methodologies: Testbench's Tyranny

xxx · Senior Joined: Jul 03, 2008 Posts: 13

As a new commer to the SoC Verificaton world at my thirties, I always maintain a gut feeling that there is something wrong with the current practice of simulation based verification. This kindom is obviously under the tyranny of Testbenches (TB). But TB is only the ``vehicle'' to transform more abstract test-cases to more concrete ones. TB is not the ``source'' of the intelligence required in verification. Rolling Eyes

Forgetting that THEMSELVES are really the source of intelligence, verification engineers seem happily tolerate and even enjoy TB's tyranny. They anxiously talk about

. The goodness of seperating HVLs with HDLs
. The OOP style in the HVLs
. The importance of a sophisticated, layered TB structure
. The comparison ``verification methodologies'' (AVM, VVM, OVM, RVM, eVM)

The EDA vendors, propagandize so many ``verification methodologies'', but to me, they are just the ``conventions'' (or even ``regulations'') to construct TBs. (And they want us to buy their ``regulations''! Twisted Evil

). And one of the nightmares for verification engineers is that the conventions always get replaced by newer conventions, so they are not real ``conventions'' at all! Shocked

What on earth is the good of ``OOP, Layered'' TB to a DUT, which is written in plain Verilog modules/VHDL processes ready for synthesis? Shouldn't we focus more on abstract test-cases for DUT rather than on how to polish the already very fanciful TB? After all, the TB is not native to a DUT.

stephenh · Posted: Fri Jul 04, 2008 8:50 am **Post subject:**

I guess the simplest answer is that no matter how intelligent the verification engineer is, there is never enough time, nor enough engineers, to allow the manual creation of all those tests for corner cases, use cases and such like.
The spangly OOP type testbenches are all trying to a greater or lesser extent to provide automation whereby the clever engineer can describe those abstract test cases and let the testbench worry about the stimulus, the checking and the coverage.
So your last paragraph actually answers your question - it's all about abstraction and productivity.

Of course a lot of things complicate matters along the way, such as rapidly growing chip complexity, which leads to the need for more advanced methodology all the time.
And of course the chip industry suffers from perhaps an excessive amount of not-invented-here syndrome, bigotry, and being a very small industry in terms of head count, each person can make a lot of noise, whether or not they have anything informed or constructive to say! Razz

I'll get off my soap-box now before I start to rant Wink

glb · Senior Joined: Feb 02, 2005 Posts: 114

xxx · Senior Joined: Jul 03, 2008 Posts: 13

I think I should put it in this way:
For any simulation practice, there are two worlds,
1. The world IN which the simulation is performed, and
2. The world FOR which the simulation is performed.

DUT, obviously is a piece of a HW in the world-2; but it is a piece of a SW in the world-1;
What is TB then? Same thing, HW in the world-2 but SW in world-1;

The purpose for simulation is definitely the behaviou in world-2, So TB, just as DUT, should be largely regarded as HW in world-2. However, the current trend is to emphasize that TB should be equiped with a lot of fancy SW things in the world-1. I am not saying it is totally worthless, but that the overhead is too big: we should focus on HW in world-2, but instead, we are trapped in the SW in the world-1.

Let us have something constructive. I think there is something missing in most verification-methodologies, at least for an SoC DUT. The missing component is the ``The SW in world-2'' -- the SW native to the DUT rather native to the simulation environment as TB is. This kind of SW does not occupy a proper position in most methodologies, but it contributes a huge part of world-2 behaviours.

Of course, we could regard this SW as the behaviour of the DUT HW. But the intrinsic flexibility of SW makes this view very awkward. Let us view world-2 SW in this way:
. It can write values to DUT components, this is its ``Control'' capability;
. It can read values from DUT components, and it can sense DUT components' behaviours in terms of interrupt/exception. These show its ``observation'' capability.

Now that this kind of SW has control and observation capability to a DUT, just like a TB has the similar stuffs on the the DUT, why cannot there be a SW-centric verification methodology similar to the TB-centric verification methodologies?

Janick · Posted: Sat Jul 05, 2008 7:32 am **Post subject:**

SW-centric verification methodologies do exist, but they are found mostly in the diagnostic and initial bring-up phases of a system.

They do not offer sufficient controllability and observability on the other interfaces of the design. For example, how could you, from a purely SW-centric view, generate relevant corner-case stimulus on the ethernet port of a networking SoC? Or verify that the outbound traffic is shaped correctly?

And how are you going to decide which test to write and how will you write them in that SW-centric methodology? Directed low-level read/write tests? or will you invest in building a fancy SW-centric TB to create a higher-level of abstraction? or to support constrained random tests?

That's what all those fancy TB methodologies help do.

Next, there is the performance issue. How are you going to run this software? A model of the processor will be really slow but co-simulation reduces the visibility of the DUT to the SW so it becomes difficult to create specific conditions. Using FPGA-based hardware prototypes or emulator boxes create a very long debug cycle which is too cumbersome for the early stages of the verification.

heritageorchard · Posted: Sat Jul 05, 2008 8:37 am **Post subject: TB pains**

Hi,

You are on the right track. The really long pole in a product is the software. So, the sooner you can get the DUT into the hands of the SW folks, the better.

I am sure you know that, as far as the SW is concerned, the world is memory reads/writes and interrupts. That's it.

For example, there are two sides to an IO protocol: the wire protocol and the SW programming interface. I call the objects that interface with the wires monitors and BFMs. I call the SW one a SFM (Software Functional Model). It's interface is memory writes and possibly interrupt handlers. Now of course, the memory interface will use a BFM. But there should be a clean separation from the code decides what registers to access versus HOW those memory accesses happen.

Now the above rant assumes that (1) the company sells a HW/SW product and the the SW can exercise the DUT fully (or the DUT is field programmable).

On to Janick's points:

xxx · Senior Joined: Jul 03, 2008 Posts: 13

Sckoarn · Posted: Mon Jul 07, 2008 7:21 am **Post subject:**

I use a primitive TB system at my work due to the tools I am given. Purely a script driven environment written in VHDL. During the past few development cycles I have come to realize that the simulation test bench is only a way to get to FPGA prototyping ASAP. Once you can get your FPGA prototype going, in what ever form, the verification on the design is accelerated greatly. Or is it that SW is now integrating and those bugs are found at an earlier time in the design cycle?

Regardless, the SW group is now developing their SW at real time or at least much better than can be done in simulation. The DUT development team can now run exhaustive iterations, be it random or directed, in much less time than is possible in simulation.

IMHO, in the long run FPGA prototyping will be the only way to go, because of the near real time or real time that can be achieved. The methodology that will win will be the one that enables easy transition from the simulation to the FPGA prototype. The FPGA prototype would house the TB as well, in this up and coming methodology, hence everything will be running as fast as possible.

I consider the software group as my customer, and it is my job to get them a working DUT as fast as possible so that SW development can be done. We all know SW development has the longest time frames, so getting they working on the actual functioning DUT ASAP is a good thing. The SW team will find bugs in the DUT, which is good, find them sooner rather than too late. The FPGA prototyping methods will be the way of the future, no matter what verification language comes to the for front.

Sckoarn ;(

dave_whipp

xxx · Senior Joined: Jul 03, 2008 Posts: 13

glb · Senior Joined: Feb 02, 2005 Posts: 114