RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases

From: Neil Korpusik <Neil.Korpusik_at_.....>
Date: Fri Nov 10 2006 - 13:43:41 PST
<forwarding email from Swapnil Dwivedi>


-------- Original Message --------
Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
Date: Fri, 10 Nov 2006 15:21:43 -0000
From: "Dwivedi, Swapnil" <swapnil_dwivedi@mentor.com>
To: "Mark Strickland \(mastrick\)" <mastrick@cisco.com>,
        "Scott, David" <david_scott@mentor.com>, <sv-ec@server.eda.org>
Cc: "Arturo Salz" <Arturo.Salz@synopsys.com>,
        "Warmke, Doug" <doug_warmke@mentor.com>


Hello Mark!

I surely do think you have a valid requirement. Users would definitively
like to have functionality which you described in your email. But I
think this feature should be part of the Coverage report scheme and onus
should lie with vendor to provide it as part of their tool's report
query system.

The "weight" option has a special meaning in LRM and is currently solely
attached to Coverage calculation. Inferring something else from it (at
different situations) would possibly have different dictations from not
only customers but vendors too. By keeping it in the confines of
Coverage calculation we can minimize that.

Now about your requirement of not showing in report the items with
"weight == 0". I would forward your requirement to our team. If you do
not mind we would be very much interested in knowing your other
requirements and would directly like to discuss with you. In our
product, "Questa", we do provide something similar in our Coverage
reporting system : "[-above <coverage_percent> | -below
<coverage_percent>]". You may use that for the moment until we come back
about your requirement.

Lets discuss your scenarios and how do we achieve your requirements.

Scenario 1) No problem here.
Scenario 2) No problem here either.

Scenario 3)
Covergroup A
A.type_option.weight = 1 (default)
A.type_option.goal = 100 Achieved = 100

Covergroup B : No Coverpoints (Say Covergroup under construction)
B.type_option.weight = 1 (default)
B.type_option.goal = 100 Achieved = 0 (no coverage objects/values => 0
achieved)

Let assume one instance of each Covergroup. Clearly coverage would be
Design: 50% (Covergroup A contribution: 100%  Covergroup B contribution:
0%)
Covergroup A coverage: 100%
Covergroup B coverage: 0%
Above confirms to one of your requirement.

Now lets see what happens if you make the "B.type_option.weight = 0" and
given my suggestion is used. Since weight is zero the concerned object
does not contribute in Coverage calculation and the new Coverage is
Design: 100% (Covergroup A contribution: 100%  Covergroup B: None)
Covergroup A coverage: 100%
Covergroup B coverage: 0%

This one almost confirms to your other requirement. You expect
"Covergroup B coverage: 100%" but it is "0%". As a user working in group
I think the results shown is perfect. When you are working on a new
Covergroup your first aim is to see the you do not un-intentionally
affect the Design Coverage (or get an ire from colleague) by suddenly
bringing down the Coverage. Your second aim is to keep seeing your
Covergroup's correct coverage so that you can keep an eye on it and keep
working on it. Of course there are other uses of "weight = 0". For
example you do not want another group's Covergroup to affect your
Coverage objectives/goals. You can create a file with hierarchical
assignments of "weight = 0" and mask their Covergroup out of coverage
calculation.

Scenario 4)
Covergroup A
A.type_option.weight = 1 (default)
A.type_option.goal = 100 Achieved = 100

Covergroup B : One coverpoint "cvp"
B.type_option.weight = 1 (default)
B.type_option.goal = 100 Achieved = 0 (cvp did not contribute)
B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage
calculation)
B.cvp.goal = 100  Achieved = 50

So the coverage is
Design: 50% (Covergroup A contribution: 100%  Covergroup B contribution:
0%)
Covergroup A coverage: 100%
Covergroup B coverage: 0% (cvp contribution : None)
Coverpoint B.cvp coverage: 50%

Now if I make "B.type_option.weight = 0" and "B.cvp.type_option.weight =
0" then (lets call this CASE A)
Design: 100% (Covergroup A contribution: 100%  Covergroup B
contribution: None)
Covergroup A coverage: 100%
Covergroup B coverage: 0%  (cvp contribution : None)
Coverpoint B.cvp coverage: 50%

Now if I make "B.type_option.weight = 0" and "B.cvp.type_option.weight =
1 (default)" (lets call this CASE B)
Design: 100%  (Covergroup A contribution: 100%  Covergroup B
contribution: None)
Covergroup A coverage: 100%
Covergroup B coverage: 50%
Coverpoint B.cvp coverage: 50%

Now as a user "CASE B" fits perfectly when working in group/multi-group
environment.


Scenario 5)
This is same as CASE A of the scenario 4.

Covergroup A
A.type_option.weight = 1 (default)
A.type_option.goal = 100 Achieved = 100

Covergroup B : One coverpoint "cvp"
B.type_option.weight = 0
B.type_option.goal = 100 Achieved = 0 (both cvp and B do not contribute)
B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage
calculation)
B.cvp.goal = 100  Achieved = 50

Design: 100% (Covergroup A contribution: 100% Covergroup B contribution:
None)
Covergroup A coverage: 100%
Covergroup B coverage: 0%
Coverpoint B.cvp coverage: 50%


As you can see "CASE B (scenario 4)" is very reasonable for almost all
the occasions. As to various other flavors which we did not discuss, the
vendor should be responsible for providing it as part of their reporting
structure.

Q)
> On a somewhat different topic, is there a concept of a choice 
> between hierarchical grading and flat grading provided to the 
> user?  To illustrate the difference, assume a covergroup with 
> 100% grade with just one bin included and another covergroup 
> with 0% grade and 99 bins included.  Assuming the groups have 
> the same weight, the hierarchical grading scheme would give an 
> overall grade of 50%, while flat grading would give 1%.  One 
> way to let the user specify may be to merge groups that all 
> have default weight as if their weight was equal to their 
> number of bins while merging explicitly weighted groups in a 
> hierarchical manner.

A) If I understand your requirement correctly then answer is no.
Covergroup coverage calculation is kind of based on name. Merging of
counts/coverage data is done with name of the object as the key. With
two different Covergroups, associating names across Covergroups etc
would soon make things go out of control.

But interestingly something similar is there within a Covergroup. Given
your example above, B.cvp.get_coverage() actually aggregates the bins at
coverpoint level with name of the bin as the key. Same is true for
Cross. So if your intention is to have two distinct sets of bins but
coverage calculation to be done on them in an aggregate fashion then use
Covergroup with ref arguments and parameters.

For example

covergroup cvg (ref cvp_expr, input int bin_size, int left_bound, int
right_bound)
{
    coverpoint cvp_expr
    {
       bins user_bin [bin_size] = {[left_cound:right_bound]};
    }
}

cvg cvg_instance1 = new(a, 10, 1, 100);
cvg cvg_instance1 = new(b, 5, 10, 25);

initial
    $display("Aggregate coverage = %f", cvg.cvp.get_coverage());

If you have further questions/queries then please feel free to send an
email to us at support@model.com

Regards,
Swapnil
++
Swapnil Dwivedi, Sr. Software Engr.
Mentor Graphics, San Jose, CA
Office : +44 (1635) 811-442  (Internal: 8210-1442)
Cell   : +44 (7919) 150-796
Home   : +1  (408)  259-1817 (VOIP)


> -----Original Message-----
> From: Mark Strickland (mastrick) [mailto:mastrick@cisco.com] 
> Sent: Thursday, November 09, 2006 9:43 PM
> To: Dwivedi, Swapnil; Scott, David; sv-ec@eda.org
> Cc: Arturo Salz; Warmke, Doug
> Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> 
> Swapnil - You could certainly define "weight=0" to mean "I 
> don't want to see this in the report" and make the algorithm 
> simple.  However, that eliminates a useful feature of "I want 
> to see this in the report but I don't want it to affect the 
> overall calculation".  One place I find this useful is when I 
> have a cross of two coverpoints, and I want to see the 
> results for the points themselves (for understanding what is 
> needed for coverage closure), but not have them contribute 
> twice (once themselves and once in the cross) to the overall result.
> 
> Dave - I agree the rule must apply at all levels and that 
> user-specified values should not be overwritten.  The 
> calculation could use a modified version of what the weight 
> field shows.  However, I am a user, not a tool creator, so I 
> think I should just state the desired user view.
> 
> Example:
> Lets say my design has two covergroups, A and B.  A has a 
> weight of 1 (by the way, that is the default if I don't say 
> anything, right?) and a grade of 100%.  B varies by scenario 
> shown below:
> 
> Scenario 1) B has a weight of 1 and one coverpoint with a 
> grade of 50% and a non-zero weight.  The overall grade for 
> this design is 75% and the grade of B is 50%.
> 
> Scenario 2) B has a weight of 0 and one coverpoint with a 
> grade of 50% and a non-zero weight.  The overall grade is 
> 100% and the grade of B is 50%.
> 
> Scenario 3) B has a weight of 1 and no coverpoints.  This is 
> a tricky one.  Most of the time, I would prefer the overall 
> grade and the grade of B to be 100%, because I do not have 
> any coverage holes.  However, occasionally it would be nice 
> to see the overall grade reflect the fact that there is a 
> covergroup I am intending to populate but have not yet done 
> so.  In this case, the overall grade would be 50% and the 
> grade of B would be 0%.  How can both modes work?  
> Option a) Always show the latter grades and make the user 
> explicitly exclude B if he/she wanted the former.  The user 
> may spend time debugging and making this fix.
> Option b) If the weight for B was unassigned in user code 
> (i.e. default value), go with the first mode.  If it was 
> assigned (even if it was assigned to 1), go with the second.  
> 
> Scenario 4) B has a weight of 1 and one coverpoint with a 
> grade of 50% and a weight of 0.  If we go with option (a) for 
> scenario (3), then the consistent behavior would be to have 
> the overall grade be 50%.  If B has a grade of 50%, we are 
> obscuring the fact that it contributed 0% to the calculation 
> for the overall grade.  If B has a grade of 0%, we are 
> obscuring the fact that there was some coverage achievement 
> in this group albeit with zero weight.  If we go with option 
> (b) for scenario (3), then the consistent behavior is to have 
> the overall grade be 100% if B had an unassigned weight or 
> 50% if it was assigned to 1.
> 
> Scenario 5) B has a weight of 0 and one coverpoint with a 
> grade of 50% and a weight of 0.  The overall grade is 100% 
> and the grade of B is 50%.
> (This scenario is very like scenario 2; there is no reason 
> that assigning the 0 weight to the coverpoint should obscure 
> its grade as viewed at the level of B.)
> 
> In all cases of scenarios 2 through 5 in which there was a 
> grade that did not contribute to the overall grade, it would 
> be ideal if some flag was set so that a GUI could highlight that fact.
> 
> On a somewhat different topic, is there a concept of a choice 
> between hierarchical grading and flat grading provided to the 
> user?  To illustrate the difference, assume a covergroup with 
> 100% grade with just one bin included and another covergroup 
> with 0% grade and 99 bins included.  Assuming the groups have 
> the same weight, the hiearchical grading scheme would give an 
> overall grade of 50%, while flat grading would give 1%.  One 
> way to let the user specify may be to merge groups that all 
> have default weight as if their weight was equal to their 
> number of bins while merging explicitly weighted groups in a 
> hierarchical manner.
> 
> Mark
> 
> -----Original Message-----
> From: Dwivedi, Swapnil [mailto:swapnil_dwivedi@mentor.com]
> Sent: Thursday, November 09, 2006 8:12 AM
> To: Scott, David; Mark Strickland (mastrick); sv-ec@eda.org
> Cc: Arturo Salz; Warmke, Doug
> Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> 
> I have a suggestion. For the case of weight being zero we can 
> take hint from Coverpoint bins.
> 
> If a Coverpoint bin, after applying the ignore and illegal 
> values, ends up with no values (they are called empty bins) 
> then we take the bin out of the coverage calculation. 
> Continuing further if all the bins in a Coverpoint are empty 
> (or there are no bins in Coverpoint) then effectively 
> Coverpoint does not have any bins and it is taken out of 
> Coverage calculation. And so on.
> 
> The point that user is giving a zero means he/she is not 
> bothered or concerned by the contribution of the object 
> (Covergroup/Coverpoint/Cross). We can apply the same rule as 
> above. A zero weight means just take the object out of 
> Coverage calculation. So if the all the Coverpoints and 
> Crosses are of zero weight then they do not exist and hence 
> Covergroup object does not (i.e. it might exist as instance 
> but does not contribute in the Coverage calculation and a 
> smart simulator might as well purge it for all purpose).
> 
> Regards,
> Swapnil
> 
> 
> 
> > -----Original Message-----
> > From: owner-sv-ec@server.eda.org
> > [mailto:owner-sv-ec@server.eda.org] On Behalf Of Scott, David
> > Sent: Thursday, November 09, 2006 2:07 AM
> > To: Mark Strickland (mastrick); sv-ec@server.eda.org
> > Cc: Arturo Salz; Warmke, Doug
> > Subject: Re: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> > 
> > I had a reaction to this one that I posted internally here 
> at Mentor 
> > last week; I've been encouraged to post it to SV-EC ...
> > 
> > 
> > If I read Mark's suggestion right, I think that is saying that 
> > all-zero weights propagate as zero weight upward, i.e., all 
> > coverpoints/crosses weighted zero imply covergroup weight = 
> 0. Is that
> 
> > right?
> > 
> > I'm worried about a couple of things.
> > 
> > One is that while this works out OK for the coverpoint/cross to 
> > covergroup level, presumably there is also weighting among 
> covergroups
> 
> > to form the $get_coverage() value. So now you have to answer the 
> > question what $get_coverage() returns if all covergroup types are 
> > weighted 0. Mark's argument can't apply at this level, I 
> don't think.
> > I'm a little bothered by an algorithm not general enough to 
> apply at 
> > all levels.
> > 
> > A more serious issue, I think, is this:
> > 
> > covergroup ct;
> > type_option.weight = 1;
> > coverpoint i { type_option.weight = 0; } coverpoint j { 
> > type_option.weight = 0; } endgroup ct cv = new;
> > 
> > initial $display(ct::type_option.weight);
> > 
> > If I have interpreted Mark's suggestion correctly, this 
> will have to 
> > display 0, in contradiction of the explicit assignment.
> > 
> > -- Dave Scott
> > 
> > 
> > Mark Strickland (mastrick) wrote:
> > > Doug/Arturo,
> > >
> > > I agree that negative weights are not needed and could be
> > considered an
> > > error.  However, I'm not sure about zero coverage when all
> > weights are
> > > zero.  I would like to be able to have two possibilities
> > for a coverage
> > > point I did not want to participate in the overall coverage
> > grade: I can
> > > still see its grade or I don't see it at all.  The weight
> > of zero is how
> > > I would achieve the first.  If a group of coverage points
> > who all have
> > > weight of zero show me a grade of zero, then I am prevented
> > from seeing
> > > the actual achievement for that group.  The solution that
> > allows me to
> > > see the grade for a group with all weights zero but does
> > not affect the
> > > overall grade is to calculate the grade as if all the
> > weights were one
> > > but then assign a weight of zero to the group grade.
> > >
> > > Mark
> > >
> > > -----Original Message-----
> > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On 
> Behalf Of 
> > > Arturo Salz
> > > Sent: Monday, October 30, 2006 2:45 PM
> > > To: Warmke, Doug; sv-ec@eda.org
> > > Subject: RE: [sv-ec] New Mantis on Coverage Calculation 
> Corner Cases
> > >
> > > Doug,
> > >
> > > I added a bugnote expressing my opinion:
> > >
> > > 1) If the sum of all weights (Wi) is zero then the coverage
> > (Cg) should
> > > yield zero.
> > >
> > > 2) A negative weight should be an error. Unsigned weights
> > are likely to
> > > just generate garbage reports and force more work upon users.
> > >
> > > 	Arturo
> > >
> > > -----Original Message-----
> > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On 
> Behalf Of 
> > > Warmke, Doug
> > > Sent: Saturday, October 28, 2006 4:22 PM
> > > To: sv-ec@eda.org
> > > Subject: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> > >
> > > Hi All,
> > >
> > > Some simple corner cases were identified in 18.10 on Coverage 
> > > Calculations.  The first is if the sum of all weights Wi in the 
> > > denominator of the Cg equation is 0.
> > > The second is what to do about negative weights.
> > >
> > > Can you please read
> > >   http://eda.org/svdb/bug_view_page.php?bug_id=0001655
> > > and express any opinions on the two corner cases?
> > >
> > > Thanks,
> > > Doug
> > >
> > >   
> > 
> > 
> 
Received on Fri Nov 10 13:43:48 2006

This archive was generated by hypermail 2.1.8 : Fri Nov 10 2006 - 13:44:12 PST