Swapnil, From the user perspective, I want to see two things (related to this topic :)): 1) I can browse the coverage results top-down and see where are my "trouble spots" of poor coverage achievement and what is my achievement of "interest only" coverage (that is not factored into my overall grade) and not get confused which is which. 2) The standard includes the fields necessary to accomplish the above and the algorithm to compute the derived values, so that I can see the same results in different tools. The tool vendor still should have latitude to take the coverage information and present it in the best way possible in their reporting tools, with colors/hierarchy/"what-if"/etc. Your proposal looks good with the exception of the following: 1) Scenario 3, when B.weight=0, seeing covergroup B grade of 0% could make me think it is a "trouble spot". The vendors could solve this by providing some kind of visual indication to me that this 0% is not included in the overall grade. 2) Scenario 4 and 5, B grade of 0% makes me think B.cvp had a grade of 0% until I probe down in the hierarchy. In a big coverage environment, I don't want to have to keep probing down to get the real data. I don't think you quite got my request for the flat grading, but perhaps I misunderstood your response. Again, I want the standard to include the grading algorithm so that I don't get different results reported from different tools. The grading algorithm, however, needs to satisfy two different needs: 1) Give me a feeling of percentage effort completed. Certainly, there are s-curves to coverage closure, but as a first cut, I'd like each new bin covered to increase this number by the same amount, so that getting the first TRUE on a TRUE/FALSE point counts the same as getting the first 5/5/5 on a three-way cross of bytes. 2) Give me a feeling of areas covered. For this purpose, I might want the first TRUE on the TRUE/FALSE to count much more because it is the only point in a covergroup and the 5/5/5 is one of many bins in a different covergroup. Currently, it sounds like only 2 is supported, but 1 is really my more common need as a report to project management. Again, the tool vendor could just throw away the grades for everything except the coverpoints and recalculate the overall grade, but the beauty of the standard to the user is that I can be sure exactly how this calculation is done. Mark -----Original Message----- From: Dwivedi, Swapnil [mailto:swapnil_dwivedi@mentor.com] Sent: Friday, November 10, 2006 10:22 AM To: Mark Strickland (mastrick); Scott, David; sv-ec@eda.org Cc: Arturo Salz; Warmke, Doug Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases Hello Mark! I surely do think you have a valid requirement. Users would definitively like to have functionality which you described in your email. But I think this feature should be part of the Coverage report scheme and onus should lie with vendor to provide it as part of their tool's report query system. The "weight" option has a special meaning in LRM and is currently solely attached to Coverage calculation. Inferring something else from it (at different situations) would possibly have different dictations from not only customers but vendors too. By keeping it in the confines of Coverage calculation we can minimize that. Now about your requirement of not showing in report the items with "weight == 0". I would forward your requirement to our team. If you do not mind we would be very much interested in knowing your other requirements and would directly like to discuss with you. In our product, "Questa", we do provide something similar in our Coverage reporting system : "[-above <coverage_percent> | -below <coverage_percent>]". You may use that for the moment until we come back about your requirement. Lets discuss your scenarios and how do we achieve your requirements. Scenario 1) No problem here. Scenario 2) No problem here either. Scenario 3) Covergroup A A.type_option.weight = 1 (default) A.type_option.goal = 100 Achieved = 100 Covergroup B : No Coverpoints (Say Covergroup under construction) B.type_option.weight = 1 (default) B.type_option.goal = 100 Achieved = 0 (no coverage objects/values => 0 achieved) Let assume one instance of each Covergroup. Clearly coverage would be Design: 50% (Covergroup A contribution: 100% Covergroup B contribution: 0%) Covergroup A coverage: 100% Covergroup B coverage: 0% Above confirms to one of your requirement. Now lets see what happens if you make the "B.type_option.weight = 0" and given my suggestion is used. Since weight is zero the concerned object does not contribute in Coverage calculation and the new Coverage is Design: 100% (Covergroup A contribution: 100% Covergroup B: None) Covergroup A coverage: 100% Covergroup B coverage: 0% This one almost confirms to your other requirement. You expect "Covergroup B coverage: 100%" but it is "0%". As a user working in group I think the results shown is perfect. When you are working on a new Covergroup your first aim is to see the you do not un-intentionally affect the Design Coverage (or get an ire from colleague) by suddenly bringing down the Coverage. Your second aim is to keep seeing your Covergroup's correct coverage so that you can keep an eye on it and keep working on it. Of course there are other uses of "weight = 0". For example you do not want another group's Covergroup to affect your Coverage objectives/goals. You can create a file with hierarchical assignments of "weight = 0" and mask their Covergroup out of coverage calculation. Scenario 4) Covergroup A A.type_option.weight = 1 (default) A.type_option.goal = 100 Achieved = 100 Covergroup B : One coverpoint "cvp" B.type_option.weight = 1 (default) B.type_option.goal = 100 Achieved = 0 (cvp did not contribute) B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage calculation) B.cvp.goal = 100 Achieved = 50 So the coverage is Design: 50% (Covergroup A contribution: 100% Covergroup B contribution: 0%) Covergroup A coverage: 100% Covergroup B coverage: 0% (cvp contribution : None) Coverpoint B.cvp coverage: 50% Now if I make "B.type_option.weight = 0" and "B.cvp.type_option.weight = 0" then (lets call this CASE A) Design: 100% (Covergroup A contribution: 100% Covergroup B contribution: None) Covergroup A coverage: 100% Covergroup B coverage: 0% (cvp contribution : None) Coverpoint B.cvp coverage: 50% Now if I make "B.type_option.weight = 0" and "B.cvp.type_option.weight = 1 (default)" (lets call this CASE B) Design: 100% (Covergroup A contribution: 100% Covergroup B contribution: None) Covergroup A coverage: 100% Covergroup B coverage: 50% Coverpoint B.cvp coverage: 50% Now as a user "CASE B" fits perfectly when working in group/multi-group environment. Scenario 5) This is same as CASE A of the scenario 4. Covergroup A A.type_option.weight = 1 (default) A.type_option.goal = 100 Achieved = 100 Covergroup B : One coverpoint "cvp" B.type_option.weight = 0 B.type_option.goal = 100 Achieved = 0 (both cvp and B do not contribute) B.cvp.type_option.weight = 0 (i.e. does not contribute in coverage calculation) B.cvp.goal = 100 Achieved = 50 Design: 100% (Covergroup A contribution: 100% Covergroup B contribution: None) Covergroup A coverage: 100% Covergroup B coverage: 0% Coverpoint B.cvp coverage: 50% As you can see "CASE B (scenario 4)" is very reasonable for almost all the occasions. As to various other flavors which we did not discuss, the vendor should be responsible for providing it as part of their reporting structure. Q) > On a somewhat different topic, is there a concept of a choice between > hierarchical grading and flat grading provided to the user? To > illustrate the difference, assume a covergroup with 100% grade with > just one bin included and another covergroup with 0% grade and 99 bins > included. Assuming the groups have the same weight, the hierarchical > grading scheme would give an overall grade of 50%, while flat grading > would give 1%. One way to let the user specify may be to merge groups > that all have default weight as if their weight was equal to their > number of bins while merging explicitly weighted groups in a > hierarchical manner. A) If I understand your requirement correctly then answer is no. Covergroup coverage calculation is kind of based on name. Merging of counts/coverage data is done with name of the object as the key. With two different Covergroups, associating names across Covergroups etc would soon make things go out of control. But interestingly something similar is there within a Covergroup. Given your example above, B.cvp.get_coverage() actually aggregates the bins at coverpoint level with name of the bin as the key. Same is true for Cross. So if your intention is to have two distinct sets of bins but coverage calculation to be done on them in an aggregate fashion then use Covergroup with ref arguments and parameters. For example covergroup cvg (ref cvp_expr, input int bin_size, int left_bound, int right_bound) { coverpoint cvp_expr { bins user_bin [bin_size] = {[left_cound:right_bound]}; } } cvg cvg_instance1 = new(a, 10, 1, 100); cvg cvg_instance1 = new(b, 5, 10, 25); initial $display("Aggregate coverage = %f", cvg.cvp.get_coverage()); If you have further questions/queries then please feel free to send an email to us at support@model.com Regards, Swapnil ++ Swapnil Dwivedi, Sr. Software Engr. Mentor Graphics, San Jose, CA Office : +44 (1635) 811-442 (Internal: 8210-1442) Cell : +44 (7919) 150-796 Home : +1 (408) 259-1817 (VOIP) > -----Original Message----- > From: Mark Strickland (mastrick) [mailto:mastrick@cisco.com] > Sent: Thursday, November 09, 2006 9:43 PM > To: Dwivedi, Swapnil; Scott, David; sv-ec@eda.org > Cc: Arturo Salz; Warmke, Doug > Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases > > Swapnil - You could certainly define "weight=0" to mean "I don't want > to see this in the report" and make the algorithm simple. However, > that eliminates a useful feature of "I want to see this in the report > but I don't want it to affect the overall calculation". One place I > find this useful is when I have a cross of two coverpoints, and I want > to see the results for the points themselves (for understanding what > is needed for coverage closure), but not have them contribute twice > (once themselves and once in the cross) to the overall result. > > Dave - I agree the rule must apply at all levels and that > user-specified values should not be overwritten. The calculation > could use a modified version of what the weight field shows. However, > I am a user, not a tool creator, so I think I should just state the > desired user view. > > Example: > Lets say my design has two covergroups, A and B. A has a weight of 1 > (by the way, that is the default if I don't say anything, right?) and > a grade of 100%. B varies by scenario shown below: > > Scenario 1) B has a weight of 1 and one coverpoint with a grade of 50% > and a non-zero weight. The overall grade for this design is 75% and > the grade of B is 50%. > > Scenario 2) B has a weight of 0 and one coverpoint with a grade of 50% > and a non-zero weight. The overall grade is 100% and the grade of B > is 50%. > > Scenario 3) B has a weight of 1 and no coverpoints. This is a tricky > one. Most of the time, I would prefer the overall grade and the grade > of B to be 100%, because I do not have any coverage holes. However, > occasionally it would be nice to see the overall grade reflect the > fact that there is a covergroup I am intending to populate but have > not yet done so. In this case, the overall grade would be 50% and the > grade of B would be 0%. How can both modes work? > Option a) Always show the latter grades and make the user explicitly > exclude B if he/she wanted the former. The user may spend time > debugging and making this fix. > Option b) If the weight for B was unassigned in user code (i.e. > default value), go with the first mode. If it was assigned (even if > it was assigned to 1), go with the second. > > Scenario 4) B has a weight of 1 and one coverpoint with a grade of 50% > and a weight of 0. If we go with option (a) for scenario (3), then > the consistent behavior would be to have the overall grade be 50%. If > B has a grade of 50%, we are obscuring the fact that it contributed 0% > to the calculation for the overall grade. If B has a grade of 0%, we > are obscuring the fact that there was some coverage achievement in > this group albeit with zero weight. If we go with option > (b) for scenario (3), then the consistent behavior is to have the > overall grade be 100% if B had an unassigned weight or 50% if it was > assigned to 1. > > Scenario 5) B has a weight of 0 and one coverpoint with a grade of 50% > and a weight of 0. The overall grade is 100% and the grade of B is > 50%. > (This scenario is very like scenario 2; there is no reason that > assigning the 0 weight to the coverpoint should obscure its grade as > viewed at the level of B.) > > In all cases of scenarios 2 through 5 in which there was a grade that > did not contribute to the overall grade, it would be ideal if some > flag was set so that a GUI could highlight that fact. > > On a somewhat different topic, is there a concept of a choice between > hierarchical grading and flat grading provided to the user? To > illustrate the difference, assume a covergroup with 100% grade with > just one bin included and another covergroup with 0% grade and 99 bins > included. Assuming the groups have the same weight, the hiearchical > grading scheme would give an overall grade of 50%, while flat grading > would give 1%. One way to let the user specify may be to merge groups > that all have default weight as if their weight was equal to their > number of bins while merging explicitly weighted groups in a > hierarchical manner. > > Mark > > -----Original Message----- > From: Dwivedi, Swapnil [mailto:swapnil_dwivedi@mentor.com] > Sent: Thursday, November 09, 2006 8:12 AM > To: Scott, David; Mark Strickland (mastrick); sv-ec@eda.org > Cc: Arturo Salz; Warmke, Doug > Subject: RE: [sv-ec] New Mantis on Coverage Calculation Corner Cases > > I have a suggestion. For the case of weight being zero we can take > hint from Coverpoint bins. > > If a Coverpoint bin, after applying the ignore and illegal values, > ends up with no values (they are called empty bins) then we take the > bin out of the coverage calculation. > Continuing further if all the bins in a Coverpoint are empty (or there > are no bins in Coverpoint) then effectively Coverpoint does not have > any bins and it is taken out of Coverage calculation. And so on. > > The point that user is giving a zero means he/she is not bothered or > concerned by the contribution of the object > (Covergroup/Coverpoint/Cross). We can apply the same rule as above. A > zero weight means just take the object out of Coverage calculation. So > if the all the Coverpoints and Crosses are of zero weight then they do > not exist and hence Covergroup object does not (i.e. it might exist as > instance but does not contribute in the Coverage calculation and a > smart simulator might as well purge it for all purpose). > > Regards, > Swapnil > > > > > -----Original Message----- > > From: owner-sv-ec@server.eda.org > > [mailto:owner-sv-ec@server.eda.org] On Behalf Of Scott, David > > Sent: Thursday, November 09, 2006 2:07 AM > > To: Mark Strickland (mastrick); sv-ec@server.eda.org > > Cc: Arturo Salz; Warmke, Doug > > Subject: Re: [sv-ec] New Mantis on Coverage Calculation Corner Cases > > > > I had a reaction to this one that I posted internally here > at Mentor > > last week; I've been encouraged to post it to SV-EC ... > > > > > > If I read Mark's suggestion right, I think that is saying that > > all-zero weights propagate as zero weight upward, i.e., all > > coverpoints/crosses weighted zero imply covergroup weight = > 0. Is that > > > right? > > > > I'm worried about a couple of things. > > > > One is that while this works out OK for the coverpoint/cross to > > covergroup level, presumably there is also weighting among > covergroups > > > to form the $get_coverage() value. So now you have to answer the > > question what $get_coverage() returns if all covergroup types are > > weighted 0. Mark's argument can't apply at this level, I > don't think. > > I'm a little bothered by an algorithm not general enough to > apply at > > all levels. > > > > A more serious issue, I think, is this: > > > > covergroup ct; > > type_option.weight = 1; > > coverpoint i { type_option.weight = 0; } coverpoint j { > > type_option.weight = 0; } endgroup ct cv = new; > > > > initial $display(ct::type_option.weight); > > > > If I have interpreted Mark's suggestion correctly, this > will have to > > display 0, in contradiction of the explicit assignment. > > > > -- Dave Scott > > > > > > Mark Strickland (mastrick) wrote: > > > Doug/Arturo, > > > > > > I agree that negative weights are not needed and could be > > considered an > > > error. However, I'm not sure about zero coverage when all > > weights are > > > zero. I would like to be able to have two possibilities > > for a coverage > > > point I did not want to participate in the overall coverage > > grade: I can > > > still see its grade or I don't see it at all. The weight > > of zero is how > > > I would achieve the first. If a group of coverage points > > who all have > > > weight of zero show me a grade of zero, then I am prevented > > from seeing > > > the actual achievement for that group. The solution that > > allows me to > > > see the grade for a group with all weights zero but does > > not affect the > > > overall grade is to calculate the grade as if all the > > weights were one > > > but then assign a weight of zero to the group grade. > > > > > > Mark > > > > > > -----Original Message----- > > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On > Behalf Of > > > Arturo Salz > > > Sent: Monday, October 30, 2006 2:45 PM > > > To: Warmke, Doug; sv-ec@eda.org > > > Subject: RE: [sv-ec] New Mantis on Coverage Calculation > Corner Cases > > > > > > Doug, > > > > > > I added a bugnote expressing my opinion: > > > > > > 1) If the sum of all weights (Wi) is zero then the coverage > > (Cg) should > > > yield zero. > > > > > > 2) A negative weight should be an error. Unsigned weights > > are likely to > > > just generate garbage reports and force more work upon users. > > > > > > Arturo > > > > > > -----Original Message----- > > > From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On > Behalf Of > > > Warmke, Doug > > > Sent: Saturday, October 28, 2006 4:22 PM > > > To: sv-ec@eda.org > > > Subject: [sv-ec] New Mantis on Coverage Calculation Corner Cases > > > > > > Hi All, > > > > > > Some simple corner cases were identified in 18.10 on Coverage > > > Calculations. The first is if the sum of all weights Wi in the > > > denominator of the Cg equation is 0. > > > The second is what to do about negative weights. > > > > > > Can you please read > > > http://eda.org/svdb/bug_view_page.php?bug_id=0001655 > > > and express any opinions on the two corner cases? > > > > > > Thanks, > > > Doug > > > > > > > > > > >Received on Tue, 21 Nov 2006 12:43:04 -0500
This archive was generated by hypermail 2.1.8 : Tue Nov 21 2006 - 09:43:45 PST