Automated verification from documentation

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.12, Item 12.0)

From: Anonymous

In my company I am trying to automate simple things like converting
the spec's register definitions to the models and to specific register
oriented tests. My life is made extremely difficult by the fact that
the architect insists he must use fancy features of MS Word, such as
colorful tables, to make the spec "readable." He also feels that
trying to automate from the spec is a dead-end, unworthy of him
changing the way he writes the spec.

My typical flow currently is to manually open the Word document, save
as text, and then run perl over it. Every spec change so far has
introduced some new quirk in the text output format that breaks my
scripts. I don't know anyway to grok the MS Word format natively, and
even if I did, the spec is not 100% self-consistent.

I am interested to hear other peoples experience with automating their
verification from the documentation, or examples of good scriptable
formats that would also be readable. Are there free or non-free tools
that can be used?

I've toyed with an idea of a in-house documentation/specification
language, as I've seen in another company, but nobody really wants to
learn a buggy in-house language. This would essentially invert the
doc/spec flow, writing code, to produce spec, instead of write spec to
produce code. I really doubt anyone spec'ing chips in my company
would do this. (sometimes it seems: those who can write code, those
who can doc/spec.)

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.12, Item 12.1)

From: Janick Bergeron

Beach Solutions (http://www.beachsolutions.com) showed a tool at DAC
that would generate functional verification code in VHDL from a
specification of the memory-mapped registers in a device. See their
website for more details.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.0)

From: David Ljung Madison

The obvious solution here is to fire the architect. :)

I'm a firm believer in documentation generation of verification.
Without it, it seems to me you have this:

1) Architect interprets spec and writes code
2) Verification engineer interprets spec and writes code
3) Compare

We find bugs in the interpretations made in #1 and #2. If those
mistakes are the same, then the bug will never be found.

Automating from documentation (assuming the automation is done
correctly! :) will avoid this problem and generally ensure correct
design to the spec.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.1)

From: Tom Verbeure

If your architect thinks that your approach to create the register
files from the spec is a dead end, your best solution might be to
replace the architect.

For our latest chip, we have used a similar approach as yours, where
we define all the registers (and its bitfields) and RAMs in a single
Excel spreadsheet. We export this spreadsheet as a CSV file (comma
separated values).

Then we use the same approach as the original poster: using a bunch of
perl scripts to create all the RTL code (address decoding, storage of
constants, ...), C header files, etc. The nice thing about Excel is
that it is a tool to make calcutions, so all the address offset
calculations are done on the fly. (Eg. changing a Ram from 10 to 11
bits will recalculate all the addresses.) All this is automated: once
you save the Excel file, our make system recreates all the RTL, header
files etc.

In the architecture spec (written in MS Word), we only used the names
of the registers, never the hardcoded values of the addresses, because
those values could change as the spreadsheet was updated later on,
which, of course, happened a lot during the development of the chip. I
am sure it must be possible to fetch these values automatically out of
the Excel document and update them automatically in the Word file
(with some Visual Basic hacking?) but we didn't bother. The software
guys were usually only using the spreadsheet anyway.

BTW: Easics (http://www.easics.com) has an online webtool to create
register definitions. The tool is driven by their own small
language. It has more or less the same features as our spreadsheet and
the software of Beach Solutions.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.2)

From: Gal Vardi

Anonymous asked about verifiable SPEC. this talk also touches the
issue: http://www.cs.indiana.edu/~sjohnson/FMCAD/Abstracts/45.html

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.3)

From: Svein Haustveit

At my former company we worked with similar ideas. We made chips with
large register based interfaces and after a period of evolution we
ended up with the following solution:

Registers are specified in a MS Word specification with a proprietary
syntax. A MS Word macro is run to extract all register definitions to
an ASCII file. This file is put under revision control and used as
input for automatic generation of:

1. VHDL procedures to write, read and verify registers for
verification.
2. TCL procedures to write, read and verify registers for FPGA
prototyping environment.
3. C procedures to write, read and verify registers in the actual
application code.
4. VHDL implementation of registers to be used in the actual
implementation.
5. Nice register map table to be added in the MS Word document for
final documentation.

Inspired by this and a growing number of pins we have added a similar
approach for pins.

Pins are specified in MS Word specification with a proprietary syntax.
A MS Word macro is run to extract all register definitions to an ASCII
file. This text file is put under revision control and used as input
for automatic generation of:

1. Toplevel entity and architecture with pads and boundary scan
2. Signoff files to ASIC vendor
3. Pin out drawing

The key motivation is SINGLE SOURCE. It is almost impossible to
maintain parallel representations manually when number of
pins/registers gets above a certain limit.

I like the concept of turning the spec into source code and what is
described above is possible without large conflict between human
readability and a strict syntax.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.4)

From: Dave Meredith

The problem with even trying to automate this sort of thing is that by
the time your spec is sufficiently detailed to be able to
automatically generate anything useful, it probably deserves to be
called a design. It's more a methodology problem - bridging the gap
between spec and design is always an issue, even without the usual
problems of keeping everything in sync.

In my previous life writing high-integrity software, we used a tool to
draw flow charts of the algorithms, control flows and so on which
could then autogenerate code from said flowcharts. Our design flow
was such that system engineers would write the high-level
specification and software engineers would take that specification and
extend it down to the level of detailed flow charts, which would then
be part of the specification and signed off by the system engineers.
The whole point was to have a smooth transition from "specification"
to "software" without gaps for bugs to creep in.

(FWIW that tool was called BEACON from ADI Systems
(http://www.adi.com). It won't be much use for development (although
you might be able to adapt it to do prototypes/behavioural models).
We used it more as a documentation tool that was guaranteed to be in
sync with the codebase.)

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.5)

From: Akiva Michelson

There are a few solutions we have used to solve these problems.

One approach is to save the Word file in HTML format which is much
easier to parse (in Perl for example). We had a good experience
converting a test list to separate files with description headers and
empty template, using this approach. Another is to do the work once,
and then look at new versions of the specification with change-bars (A
feature in Word for tracking changes)

If the above approaches don't work, you may be able to get a secretary
to do it for you, and the other alternative is to have each person on
the team take a section and simply "get it done". I strongly suggest
having each person on the team take a part in it, rather then create a
stigma of certain work for certain people.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.6)

From: Anonymous

I have tried to push this issue at my organization too. I suggested
that the architects write their specs using an XML-driven editor -
there are some fairly WYSIWYG editors that produce XML (e.g. Epic by
Arbortext).

XML is supposed to solve the problem of lack of structure on the web
(in HTML). Writing your architecture in XML (under the appropriate
standards) also enables electronic component intercharge which is a
growing fad.

To use such a scheme, one need three elements:

* A DTD (or grammer or template) that defines the structural elements
in a document: chip, pins / pads, modules, registers (with masks),
etc.
* Style-sheets (or mappers) that can map such XML documents into
marked-up documents on the one hand, and design data - such as basic
definitions in DV models - on the other hand.
* Documents, generated by the XML-driven editor. These could be
managed on a database.

For example, the registers in the document could look something like:

...
id="R025"
name="bus_config"
width="word"
offset="0xfbfe"
...
>
...
>
...

The editor I saw was not as mature as Word or Framemaker, and does not
handle graphics as well (one needs to import graphics into the
document). I assume these editors will improve over time - because of
the popularity of XML.

Note that (as its name implies) "Specman" was planned to be used by
architects for SPECification in the E language (E - English). It did
not succeed with architects (similarly to formal specification
tools). The basic problem you and I have is that the architect is
being asked to invest effort in order to make the verification
easier. You need help from your management ...

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.7)

From: Nicholas Jamba

I would strongly suggest automation for code that deals with
registers. There are at least three groups that use the register
fields and addressing, design, software (drivers) and
verification. Whenever there is a change in the registers they all
have to go in and modify their code.

It does not matter how the registers are specified as long as it is a
consistent language between all designers. From one you should be
able to create all the other files (verilog, c, e, vera, word, pdf)
with your favorite parsing tool. In the verification area you should
be able to create the register access infrastructure for all the tests
and all the registers tests themselves (r/w, read only, etc). We have
gone as far as to include all of the accessible memory including
tables and such. A tables is just a repeated register format with a
size and offset.

A lot of complaints may come from the designers at first, but after
they see how easy it is and the time saved they will all trickle over.
It has to start somewhere though!

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 2.13, Item 4.8)

From: Mark Gonzales

At a previous company I worked for, they used a set of Word macros to
write out tables from a Word document in a perl-readable format for
processing into testbench code.

This seemed to work pretty well.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.12, Item 3.0)

From: Scott Borisch

Unlike many contributors to Mr. Bergeron's website (I'm guessing), I
work with FPGAs instead of ASICs. Instead of designing ASICs meant
for use on many designs, the FPGAs I design with are generally used on
a small number of boards, and the functionality required is usually
dictated by the requirements of the analog circuitry on the product
(board level design).

Thus, I'm usually the one to specify the register map
(digital design engineer) and I came up with a variation on
the techniques discussed in this thread:

I specify the register map in a VHDL data structure I created: it
represents an array of registers, with each register containing an
array of bitfields. Then I have (non-synthesizeable) VHDL files that
act as text generators -- one "generator" for each output file
produced. So far, I have an HTML register map generator
(documentation), C header file generator, and several others. The
main problem was that the structure ended up being so complex that
none of our synthesizers (including Synplicity Pro) could deal with
the data structure if I tried to include it for synthesis (ModelSim
has no problem using it in simulation).

The answer was to simply create a simplified version of the register
map information in a (synthesizeable) VHDL package file. This also
has the advantage of separating synthesis from the auto-generation of
the other files -- if a certain data structure breaks the synthesizer,
at least the entire design flow is not broken.

The final advantage is that no parser is even required -- the data is
already in the form of VHDL constants, and hence the file generators
merely need to loop through (each register) in the constant that
represents the array of registers.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.13, Item 4.0)

From: Jeff Koehler

At Fabric Networks, our ASIC designer was using Chuck Benz's csrGen

http://www.chuckbenz.com/asics/
http://www.chuckbenz.com/asics/#csrGen_-_generate_verilog_RTL_code_for

It's a perl script to automate the synthesis of the processor
interface (not surprising, since the designer was Chuck Benz!).
This gave me an idea -- I used perl's Data::Dumper package to write
out the internal database as it was running.

Another program (not GPL or public [yet]), csrGen_2_header read
this in and created several output files:

1) A pretty text file, column formatted, etc.
2) An HTML file (even prettier)
3) A Verilog header file (`defines, a lot of them)
4) A Vera class stub, which #included the `defines above

This was all based on make so it was painless to check out a new CVS
tree and rebuild the files.

The Verilog header allowed us to use mnemonics for register offsets
when writing crude testbenches.

The Vera class was more sophisticated -- it propagated attributes into
a Vera database class that allowed us to:

1) Reference individual bitfields by mnemonic: e.g.
the REVISION register is: { [16:8] DEVICE_ID , [7:0] DEVICE_REV }
and we could reference it by DEVICE_REV.

a) This prevented us from having any dependency on what register
a bitfield resided in; change at will, designer! (he did ...)
b) An additional argument on the Vera access class was 'port',
so if a register was repeated n times, you could specify
"read(PORT_STATE,n)" without regard to the address map.
c) An additional layer allowed for an ASIC instance argument.

2) Carry all of the register's attributes into the HVL world,
such as read-only, write-1-to-clear, clear-on-read, etc.
a) A Vera base class was developed that could automate
testing processor address uniqueness, data retention,
etc. Since all registers that existed were in the
database, it was easy to create a general class that
iterated over all of them. No forgotten fields.
b) A cacheing class could more easily optimize DV
processor access by knowing a bit was read-only,
write through (e.g. for write-1-to-clear).

Note these attributes were applied when the hardware was
synthesized, because they were declared to csrGen.

3) Functional coverage automation was enhanced because we
could record that a particular BITFIELD was read, and give
coverage to only the fields actually referenced -- and not
other bits in a register that were read simply for their
juxtaposition in the register map.

a) We created functional bitfield coverage reports to
ensure the verification process was exercising all
individual fields, not just registers.
b) By using the actual register definition source for the
coverage list, we easily noticed when a new bitfield was
added that had no verification associated with it.

Note the difference between Verilog toggle coverage and bitfield
access coverage -- the former does not attempt to correlate the
value with a verification action; the latter does (so long as the
test that references the bitfield is useful).

We found this to be very useful and I'm looking forward to using this
on the next ASIC, because it took a bit of work to develop. Our only
requirement is that designers use csrGen (or at least mirror their
designs with a csrGen definitions file, but then the correct-by-design
value is lost).

The support structure of classes that wrapped up this database could
be made portable because they simply take this database as a handle.

Execution time was only a few seconds.

- Jeff Koehler, Fabric Networks, Inc.