RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases

Hello Mark,

We discovered a previously passed proposal that modifies this
area of the LRM:

http://www.verilog.org/svdb/bug_view_page.php?bug_id=0001237

We are going to have to re-do our proposal for 1655 so that
it doesn't conflict with the text of 1237.  Stay tuned.

Regards,
Doug

> -----Original Message-----
> From: Mark Strickland (mastrick) [mailto:mastrick@cisco.com] 
> Sent: Tuesday, November 21, 2006 9:43 AM
> To: Dwivedi, Swapnil; Scott, David; sv-ec@eda.org
> Cc: Arturo Salz; Warmke, Doug
> Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> 
> Swapnil,
> 
> From the user perspective, I want to see two things (related to this
> topic :)):
> 
> 1) I can browse the coverage results top-down and see where are my
> "trouble spots" of poor coverage achievement and what is my 
> achievement
> of "interest only" coverage (that is not factored into my 
> overall grade)
> and not get confused which is which.
> 
> 2) The standard includes the fields necessary to accomplish the above
> and the algorithm to compute the derived values, so that I can see the
> same results in different tools.
> 
> The tool vendor still should have latitude to take the coverage
> information and present it in the best way possible in their reporting
> tools, with colors/hierarchy/"what-if"/etc.
> 
> Your proposal looks good with the exception of the following:
> 
> 1) Scenario 3, when B.weight=0, seeing covergroup B grade of 0% could
> make me think it is a "trouble spot".  The vendors could solve this by
> providing some kind of visual indication to me that this 0% is not
> included in the overall grade.
> 
> 2) Scenario 4 and 5, B grade of 0% makes me think B.cvp had a grade of
> 0% until I probe down in the hierarchy.  In a big coverage 
> environment,
> I don't want to have to keep probing down to get the real data.
> 
> I don't think you quite got my request for the flat grading, 
> but perhaps
> I misunderstood your response.  Again, I want the standard to include
> the grading algorithm so that I don't get different results reported
> from different tools.  The grading algorithm, however, needs 
> to satisfy
> two different needs:
> 
> 1) Give me a feeling of percentage effort completed.  Certainly, there
> are s-curves to coverage closure, but as a first cut, I'd 
> like each new
> bin covered to increase this number by the same amount, so 
> that getting
> the first TRUE on a TRUE/FALSE point counts the same as getting the
> first 5/5/5 on a three-way cross of bytes.  
> 
> 2) Give me a feeling of areas covered.  For this purpose, I might want
> the first TRUE on the TRUE/FALSE to count much more because it is the
> only point in a covergroup and the 5/5/5 is one of many bins in a
> different covergroup.
> 
> Currently, it sounds like only 2 is supported, but 1 is really my more
> common need as a report to project management.
> 
> Again, the tool vendor could just throw away the grades for everything
> except the coverpoints and recalculate the overall grade, but 
> the beauty
> of the standard to the user is that I can be sure exactly how this
> calculation is done.
> 
> Mark
> 
> 
> -----Original Message-----
> From: Dwivedi, Swapnil [mailto:swapnil_dwivedi@mentor.com] 
> Sent: Friday, November 10, 2006 10:22 AM
> To: Mark Strickland (mastrick); Scott, David; sv-ec@eda.org
> Cc: Arturo Salz; Warmke, Doug
> Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> 
> Hello Mark!
> 
> I surely do think you have a valid requirement. Users would 
> definitively
> like to have functionality which you described in your email. But I
> think this feature should be part of the Coverage report 
> scheme and onus
> should lie with vendor to provide it as part of their tool's report
> query system.
> 
> The "weight" option has a special meaning in LRM and is 
> currently solely
> attached to Coverage calculation. Inferring something else from it (at
> different situations) would possibly have different 
> dictations from not
> only customers but vendors too. By keeping it in the confines of
> Coverage calculation we can minimize that.
> 
> Now about your requirement of not showing in report the items with
> "weight == 0". I would forward your requirement to our team. If you do
> not mind we would be very much interested in knowing your other
> requirements and would directly like to discuss with you. In our
> product, "Questa", we do provide something similar in our Coverage
> reporting system : "[-above <coverage_percent> | -below
> <coverage_percent>]". You may use that for the moment until 
> we come back
> about your requirement.
> 
> Lets discuss your scenarios and how do we achieve your requirements.
> 
> Scenario 1) No problem here.
> Scenario 2) No problem here either.
> 
> Scenario 3)
> Covergroup A
> A.type_option.weight = 1 (default)
> A.type_option.goal = 100 Achieved = 100
> 
> Covergroup B : No Coverpoints (Say Covergroup under construction)
> B.type_option.weight = 1 (default) B.type_option.goal = 100 
> Achieved = 0
> (no coverage objects/values => 0
> achieved)
> 
> Let assume one instance of each Covergroup. Clearly coverage would be
> Design: 50% (Covergroup A contribution: 100%  Covergroup B 
> contribution:
> 0%)
> Covergroup A coverage: 100%
> Covergroup B coverage: 0%
> Above confirms to one of your requirement.
> 
> Now lets see what happens if you make the 
> "B.type_option.weight = 0" and
> given my suggestion is used. Since weight is zero the concerned object
> does not contribute in Coverage calculation and the new Coverage is
> Design: 100% (Covergroup A contribution: 100%  Covergroup B: None)
> Covergroup A coverage: 100% Covergroup B coverage: 0%
> 
> This one almost confirms to your other requirement. You expect
> "Covergroup B coverage: 100%" but it is "0%". As a user 
> working in group
> I think the results shown is perfect. When you are working on a new
> Covergroup your first aim is to see the you do not un-intentionally
> affect the Design Coverage (or get an ire from colleague) by suddenly
> bringing down the Coverage. Your second aim is to keep seeing your
> Covergroup's correct coverage so that you can keep an eye on 
> it and keep
> working on it. Of course there are other uses of "weight = 0". For
> example you do not want another group's Covergroup to affect your
> Coverage objectives/goals. You can create a file with hierarchical
> assignments of "weight = 0" and mask their Covergroup out of coverage
> calculation.
> 
> Scenario 4)
> Covergroup A
> A.type_option.weight = 1 (default)
> A.type_option.goal = 100 Achieved = 100
> 
> Covergroup B : One coverpoint "cvp"
> B.type_option.weight = 1 (default)
> B.type_option.goal = 100 Achieved = 0 (cvp did not contribute)
> B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage
> calculation)
> B.cvp.goal = 100  Achieved = 50
> 
> So the coverage is
> Design: 50% (Covergroup A contribution: 100%  Covergroup B 
> contribution:
> 0%)
> Covergroup A coverage: 100%
> Covergroup B coverage: 0% (cvp contribution : None) Coverpoint B.cvp
> coverage: 50%
> 
> Now if I make "B.type_option.weight = 0" and 
> "B.cvp.type_option.weight =
> 0" then (lets call this CASE A)
> Design: 100% (Covergroup A contribution: 100%  Covergroup B
> contribution: None)
> Covergroup A coverage: 100%
> Covergroup B coverage: 0%  (cvp contribution : None) Coverpoint B.cvp
> coverage: 50%
> 
> Now if I make "B.type_option.weight = 0" and 
> "B.cvp.type_option.weight =
> 1 (default)" (lets call this CASE B)
> Design: 100%  (Covergroup A contribution: 100%  Covergroup B
> contribution: None)
> Covergroup A coverage: 100%
> Covergroup B coverage: 50%
> Coverpoint B.cvp coverage: 50%
> 
> Now as a user "CASE B" fits perfectly when working in 
> group/multi-group
> environment.
> 
> 
> Scenario 5)
> This is same as CASE A of the scenario 4.
> 
> Covergroup A
> A.type_option.weight = 1 (default)
> A.type_option.goal = 100 Achieved = 100
> 
> Covergroup B : One coverpoint "cvp"
> B.type_option.weight = 0
> B.type_option.goal = 100 Achieved = 0 (both cvp and B do not 
> contribute)
> B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage
> calculation)
> B.cvp.goal = 100  Achieved = 50
> 
> Design: 100% (Covergroup A contribution: 100% Covergroup B 
> contribution:
> None)
> Covergroup A coverage: 100%
> Covergroup B coverage: 0%
> Coverpoint B.cvp coverage: 50%
> 
> 
> As you can see "CASE B (scenario 4)" is very reasonable for almost all
> the occasions. As to various other flavors which we did not 
> discuss, the
> vendor should be responsible for providing it as part of 
> their reporting
> structure.
> 
> Q) 
> > On a somewhat different topic, is there a concept of a 
> choice between 
> > hierarchical grading and flat grading provided to the user?  To 
> > illustrate the difference, assume a covergroup with 100% grade with 
> > just one bin included and another covergroup with 0% grade 
> and 99 bins
> 
> > included.  Assuming the groups have the same weight, the 
> hierarchical 
> > grading scheme would give an overall grade of 50%, while 
> flat grading 
> > would give 1%.  One way to let the user specify may be to 
> merge groups
> 
> > that all have default weight as if their weight was equal to their 
> > number of bins while merging explicitly weighted groups in a 
> > hierarchical manner.
> 
> A) If I understand your requirement correctly then answer is no.
> Covergroup coverage calculation is kind of based on name. Merging of
> counts/coverage data is done with name of the object as the key. With
> two different Covergroups, associating names across Covergroups etc
> would soon make things go out of control.
> 
> But interestingly something similar is there within a 
> Covergroup. Given
> your example above, B.cvp.get_coverage() actually aggregates 
> the bins at
> coverpoint level with name of the bin as the key. Same is true for
> Cross. So if your intention is to have two distinct sets of bins but
> coverage calculation to be done on them in an aggregate 
> fashion then use
> Covergroup with ref arguments and parameters.
> 
> For example
> 
> covergroup cvg (ref cvp_expr, input int bin_size, int left_bound, int
> right_bound)
> {
>     coverpoint cvp_expr
>     {
>        bins user_bin [bin_size] = {[left_cound:right_bound]};
>     }
> }
> 
> cvg cvg_instance1 = new(a, 10, 1, 100);
> cvg cvg_instance1 = new(b, 5, 10, 25);
> 
> initial
>     $display("Aggregate coverage = %f", cvg.cvp.get_coverage());
> 
> If you have further questions/queries then please feel free to send an
> email to us at support@model.com
> 
> Regards,
> Swapnil
> ++
> Swapnil Dwivedi, Sr. Software Engr.
> Mentor Graphics, San Jose, CA
> Office : +44 (1635) 811-442  (Internal: 8210-1442)
> Cell   : +44 (7919) 150-796
> Home   : +1  (408)  259-1817 (VOIP)
> 
> 
> > -----Original Message-----
> > From: Mark Strickland (mastrick) [mailto:mastrick@cisco.com]
> > Sent: Thursday, November 09, 2006 9:43 PM
> > To: Dwivedi, Swapnil; Scott, David; sv-ec@eda.org
> > Cc: Arturo Salz; Warmke, Doug
> > Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> > 
> > Swapnil - You could certainly define "weight=0" to mean "I 
> don't want 
> > to see this in the report" and make the algorithm simple.  However, 
> > that eliminates a useful feature of "I want to see this in 
> the report 
> > but I don't want it to affect the overall calculation".  
> One place I 
> > find this useful is when I have a cross of two coverpoints, 
> and I want
> 
> > to see the results for the points themselves (for 
> understanding what 
> > is needed for coverage closure), but not have them contribute twice 
> > (once themselves and once in the cross) to the overall result.
> > 
> > Dave - I agree the rule must apply at all levels and that 
> > user-specified values should not be overwritten.  The calculation 
> > could use a modified version of what the weight field 
> shows.  However,
> 
> > I am a user, not a tool creator, so I think I should just state the 
> > desired user view.
> > 
> > Example:
> > Lets say my design has two covergroups, A and B.  A has a 
> weight of 1 
> > (by the way, that is the default if I don't say anything, 
> right?) and 
> > a grade of 100%.  B varies by scenario shown below:
> > 
> > Scenario 1) B has a weight of 1 and one coverpoint with a 
> grade of 50%
> 
> > and a non-zero weight.  The overall grade for this design 
> is 75% and 
> > the grade of B is 50%.
> > 
> > Scenario 2) B has a weight of 0 and one coverpoint with a 
> grade of 50%
> 
> > and a non-zero weight.  The overall grade is 100% and the 
> grade of B 
> > is 50%.
> > 
> > Scenario 3) B has a weight of 1 and no coverpoints.  This 
> is a tricky 
> > one.  Most of the time, I would prefer the overall grade 
> and the grade
> 
> > of B to be 100%, because I do not have any coverage holes.  
> However, 
> > occasionally it would be nice to see the overall grade reflect the 
> > fact that there is a covergroup I am intending to populate but have 
> > not yet done so.  In this case, the overall grade would be 
> 50% and the
> 
> > grade of B would be 0%.  How can both modes work?
> > Option a) Always show the latter grades and make the user 
> explicitly 
> > exclude B if he/she wanted the former.  The user may spend time 
> > debugging and making this fix.
> > Option b) If the weight for B was unassigned in user code (i.e. 
> > default value), go with the first mode.  If it was assigned 
> (even if 
> > it was assigned to 1), go with the second.
> > 
> > Scenario 4) B has a weight of 1 and one coverpoint with a 
> grade of 50%
> 
> > and a weight of 0.  If we go with option (a) for scenario (3), then 
> > the consistent behavior would be to have the overall grade 
> be 50%.  If
> 
> > B has a grade of 50%, we are obscuring the fact that it 
> contributed 0%
> 
> > to the calculation for the overall grade.  If B has a grade 
> of 0%, we 
> > are obscuring the fact that there was some coverage achievement in 
> > this group albeit with zero weight.  If we go with option
> > (b) for scenario (3), then the consistent behavior is to have the 
> > overall grade be 100% if B had an unassigned weight or 50% 
> if it was 
> > assigned to 1.
> > 
> > Scenario 5) B has a weight of 0 and one coverpoint with a 
> grade of 50%
> 
> > and a weight of 0.  The overall grade is 100% and the grade of B is 
> > 50%.
> > (This scenario is very like scenario 2; there is no reason that 
> > assigning the 0 weight to the coverpoint should obscure its 
> grade as 
> > viewed at the level of B.)
> > 
> > In all cases of scenarios 2 through 5 in which there was a 
> grade that 
> > did not contribute to the overall grade, it would be ideal if some 
> > flag was set so that a GUI could highlight that fact.
> > 
> > On a somewhat different topic, is there a concept of a 
> choice between 
> > hierarchical grading and flat grading provided to the user?  To 
> > illustrate the difference, assume a covergroup with 100% grade with 
> > just one bin included and another covergroup with 0% grade 
> and 99 bins
> 
> > included.  Assuming the groups have the same weight, the 
> hiearchical 
> > grading scheme would give an overall grade of 50%, while 
> flat grading 
> > would give 1%.  One way to let the user specify may be to 
> merge groups
> 
> > that all have default weight as if their weight was equal to their 
> > number of bins while merging explicitly weighted groups in a 
> > hierarchical manner.
> > 
> > Mark
> > 
> > -----Original Message-----
> > From: Dwivedi, Swapnil [mailto:swapnil_dwivedi@mentor.com]
> > Sent: Thursday, November 09, 2006 8:12 AM
> > To: Scott, David; Mark Strickland (mastrick); sv-ec@eda.org
> > Cc: Arturo Salz; Warmke, Doug
> > Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> > 
> > I have a suggestion. For the case of weight being zero we can take 
> > hint from Coverpoint bins.
> > 
> > If a Coverpoint bin, after applying the ignore and illegal values, 
> > ends up with no values (they are called empty bins) then we 
> take the 
> > bin out of the coverage calculation.
> > Continuing further if all the bins in a Coverpoint are 
> empty (or there
> 
> > are no bins in Coverpoint) then effectively Coverpoint does 
> not have 
> > any bins and it is taken out of Coverage calculation. And so on.
> > 
> > The point that user is giving a zero means he/she is not 
> bothered or 
> > concerned by the contribution of the object 
> > (Covergroup/Coverpoint/Cross). We can apply the same rule 
> as above. A 
> > zero weight means just take the object out of Coverage 
> calculation. So
> 
> > if the all the Coverpoints and Crosses are of zero weight 
> then they do
> 
> > not exist and hence Covergroup object does not (i.e. it 
> might exist as
> 
> > instance but does not contribute in the Coverage calculation and a 
> > smart simulator might as well purge it for all purpose).
> > 
> > Regards,
> > Swapnil
> > 
> > 
> > 
> > > -----Original Message-----
> > > From: owner-sv-ec@server.eda.org
> > > [mailto:owner-sv-ec@server.eda.org] On Behalf Of Scott, David
> > > Sent: Thursday, November 09, 2006 2:07 AM
> > > To: Mark Strickland (mastrick); sv-ec@server.eda.org
> > > Cc: Arturo Salz; Warmke, Doug
> > > Subject: Re: [sv-ec] New Mantis on Coverage Calculation 
> Corner Cases
> > > 
> > > I had a reaction to this one that I posted internally here
> > at Mentor
> > > last week; I've been encouraged to post it to SV-EC ...
> > > 
> > > 
> > > If I read Mark's suggestion right, I think that is saying that 
> > > all-zero weights propagate as zero weight upward, i.e., all 
> > > coverpoints/crosses weighted zero imply covergroup weight =
> > 0. Is that
> > 
> > > right?
> > > 
> > > I'm worried about a couple of things.
> > > 
> > > One is that while this works out OK for the coverpoint/cross to 
> > > covergroup level, presumably there is also weighting among
> > covergroups
> > 
> > > to form the $get_coverage() value. So now you have to answer the 
> > > question what $get_coverage() returns if all covergroup types are 
> > > weighted 0. Mark's argument can't apply at this level, I
> > don't think.
> > > I'm a little bothered by an algorithm not general enough to
> > apply at
> > > all levels.
> > > 
> > > A more serious issue, I think, is this:
> > > 
> > > covergroup ct;
> > > type_option.weight = 1;
> > > coverpoint i { type_option.weight = 0; } coverpoint j { 
> > > type_option.weight = 0; } endgroup ct cv = new;
> > > 
> > > initial $display(ct::type_option.weight);
> > > 
> > > If I have interpreted Mark's suggestion correctly, this
> > will have to
> > > display 0, in contradiction of the explicit assignment.
> > > 
> > > -- Dave Scott
> > > 
> > > 
> > > Mark Strickland (mastrick) wrote:
> > > > Doug/Arturo,
> > > >
> > > > I agree that negative weights are not needed and could be
> > > considered an
> > > > error.  However, I'm not sure about zero coverage when all
> > > weights are
> > > > zero.  I would like to be able to have two possibilities
> > > for a coverage
> > > > point I did not want to participate in the overall coverage
> > > grade: I can
> > > > still see its grade or I don't see it at all.  The weight
> > > of zero is how
> > > > I would achieve the first.  If a group of coverage points
> > > who all have
> > > > weight of zero show me a grade of zero, then I am prevented
> > > from seeing
> > > > the actual achievement for that group.  The solution that
> > > allows me to
> > > > see the grade for a group with all weights zero but does
> > > not affect the
> > > > overall grade is to calculate the grade as if all the
> > > weights were one
> > > > but then assign a weight of zero to the group grade.
> > > >
> > > > Mark
> > > >
> > > > -----Original Message-----
> > > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On
> > Behalf Of
> > > > Arturo Salz
> > > > Sent: Monday, October 30, 2006 2:45 PM
> > > > To: Warmke, Doug; sv-ec@eda.org
> > > > Subject: RE: [sv-ec] New Mantis on Coverage Calculation
> > Corner Cases
> > > >
> > > > Doug,
> > > >
> > > > I added a bugnote expressing my opinion:
> > > >
> > > > 1) If the sum of all weights (Wi) is zero then the coverage
> > > (Cg) should
> > > > yield zero.
> > > >
> > > > 2) A negative weight should be an error. Unsigned weights
> > > are likely to
> > > > just generate garbage reports and force more work upon users.
> > > >
> > > > 	Arturo
> > > >
> > > > -----Original Message-----
> > > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On
> > Behalf Of
> > > > Warmke, Doug
> > > > Sent: Saturday, October 28, 2006 4:22 PM
> > > > To: sv-ec@eda.org
> > > > Subject: [sv-ec] New Mantis on Coverage Calculation Corner Cases
> > > >
> > > > Hi All,
> > > >
> > > > Some simple corner cases were identified in 18.10 on Coverage 
> > > > Calculations.  The first is if the sum of all weights Wi in the 
> > > > denominator of the Cg equation is 0.
> > > > The second is what to do about negative weights.
> > > >
> > > > Can you please read
> > > >   http://eda.org/svdb/bug_view_page.php?bug_id=0001655
> > > > and express any opinions on the two corner cases?
> > > >
> > > > Thanks,
> > > > Doug
> > > >
> > > >   
> > > 
> > > 
> > 
>