RE: [sv-ec] Completing my coverage thought...

From: Arturo Salz <Arturo.Salz_at_.....>
Date: Wed Jan 18 2006 - 13:14:51 PST
I'll try to clarify.

 

In my previous response to Swapnajit I was trying to make it clear that
ignore bins 

do not affect the "overall coverage space". What I meant (and I believe
that was the 

gist of the question) is that an "ignore bins" specification does not
affect how bins 

are computed. Otherwise, one might consider removing the ignored values
from the 

space and then distribute the remaining (non ignored) values into the
resulting bins.

 

Until the recent discussion, what we now call "empty bins" was not
considered special 

in any way (other than the fact that it could never be covered). We are
now refining our 

thinking to remove the contribution to coverage of these empty bins.
This new thinking 

does indeed change the way coverage is computed from the way I had
described it in 

my earlier response to Swapnajit.

 

Our latest thinking is that empty bins should not affect coverage. Gord
has identified 

one corner case (a coverpoint with all empty bins) and proposed a
solution, but I don't 

believe we have consensus on how to deal with it. My thinking is that
there are four 

alternatives (the first three were proposed by Gord):

   1) illegal

   2) 100% coverage

   3) 0% coverage

   4) eliminate from coverage

 

The reason for the 4th option is that an empty coverpoint can be
simultaneously considered 

as being trivially covered and not covered. In other words, we are
devising semantics to 

resolve the mathematical indeterminacy of dividing zero by zero, which
could indeed be 0 or 1.

Since I don't know how we could apply L'Hopital's rule to coverage, I
believe the best thing is 

to remove it from coverage. By that I mean the following:

  1) An empty coverpoint (one that consists only of empty bins) does not
affect the coverage

     score of the covergoup, that means, remove it from consideration
from both the numerator

     and the denominator.

  2) An empty covergroup (one that consists entirely of empty
coverpoints) does not affect

      overall coverage score.

A reporting tool should probably show these empty cover points/groups
differently. I think that

empty coverpoints are equivalent to unreachable states in state coverage
(or unreachable lines

in line coverage), and likewise, they should not modify the coverage
score.

 

            Arturo

 

-----Original Message-----
From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On Behalf Of
Gordon Vreugdenhil
Sent: Wednesday, January 18, 2006 7:18 AM
To: Swapnajit Chakraborti
Cc: sv-ec@eda.org
Subject: Re: [sv-ec] Completing my coverage thought...

 

 

 

Swapnajit Chakraborti wrote:

 

>  

> Gord,

> 

> Thanks for clarifying the context. According to you, one scenario that

> can result "empty" bins is the interaction of ignore_bins with other

> bins.

> Can you please elaborate a little more on this? 

> Do you mean that in the following example, bin b1 will be interpreted
as

> 

> an "empty" bin? 

 

Based on the recent discussion, the answer is yes.

 

 

> coverpoint cp1 {

>    bins b1 = {0, 1, 2};

>    bins b2[] = {0, 3, [7:15] };

>    ignore_bins ig1 = {0, 1, 2};

> } 

> 

> If that is the case, then ignore_bins is actually affecting the total

> coverage

> space. Is that the desired behavior? In this context, I am attaching

> below

> an earlier discussion, where Arturo wrote to one of my queries that

> ignore_bins will not

> affect the total coverage space.

 

Arturo, can you respond on this?  Based on what we discussed,

it seems that the coverage in the example that Swapnajit had

asked about earlier should in fact be 100%.

 

Gord.

 

 

> 

> Arturo Salz wrote:

> 

>>Swapnajit,

>> 

>>An ignored coverbin does not modify the coverage space, it only

> 

> indicates

> 

>>that the specified bins not be counted.

>>Thus, your example would do the following:

>>    - Create 7 auto-bins (0..6).

>>    - Bins 0, 1, and 2 never take a hit.

>>And, if the sampled "a" takes on all values 0-7, the coverage is 4/7

> 

> => 57.1%

> 

>>    Answer

> 

> 

> And here is the relevant example referred in Arturo's comment:

> 

> 

>>  reg [2:0] a;

>>  coverpoint a {

>>     ignore_bins ig_bins = {0, 1, 2};

>>  }

> 

> 

> Please let me know your comments on this. I do agree that if
ignore_bins

> does not

> affect total coverage space, we can never reach 100% coverage. These

> points

> need to be clarified.

> 

> The other point you mentioned that "empty" bins can result if there
are

> more 

> bins than values. I found this in LRM as well and clearly there is no

> mention

> whether to include them in coverage calculation. But according to me,

> since 

> empty bins resulted from user specification, we should consider them
for

> coverage calculation. So, if user specifies following:

> 

> reg [2:0] a;

> coverpoint a {

>   bins b1[10] = {[0:7]};

> }

> 

> And values reached during simulation are 0:7, the coverage will be

> reported as 8/10, considering the "empty" bins, namely b1[8] and b1[9]

> in the total.

>  

> Thx,

> Swapnajit.

> 

> 

> 

> 

>>-----Original Message-----

>>From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On Behalf Of 

>>Gordon Vreugdenhil

>>Sent: Tuesday, January 17, 2006 11:18 PM

>>To: Swapnajit Chakraborti

>>Cc: Arturo Salz; SV_EC List

>>Subject: Re: [sv-ec] Completing my coverage thought...

>> 

>>Swapnajit,

>> 

>>The discussion we were having was a continuation of the SV-EC 

>>conference call which is why you misunderstood this.

>> 

>>By "empty bins" here we did not mean "bins that were never covered"
but

> 

> 

>>rather "bins with no elements in their value set".

>>This arises due to the interaction of ignore_bins and/or having more 

>>bins than values.

>> 

>>The LRM has not defined whether bins that have no elements in their 

>>value set are supposed to be included in the computation.

>>Arturo and I are in agreement that they should not.  As I noted in my 

>>follow-up email, there is still edge case behavior to discuss - the 

>>situation in which there are no bins with values in their value sets.

>> 

>>Gord.

>> 

>> 

>> 

>>Swapnajit Chakraborti wrote:

>> 

>> 

>>>Hi Arturo & Gord,

>>> 

>>>As per LRM, coverage for a coverpoint (section 18.10.1) is

>> 

>>defined as:

>> 

>>>  num_covered_bins/num_all_defined_bins

>>> 

>>>Again, num_covered_bins is actually the number of bins for which 

>>>at_least count is reached (count >= at_least).

>>> 

>>>Now, in your discussion you have mentioned "num_non_empty_bins".

>>>I am assuming the following definition of "num_non_empty_bins".

>>> 

>>>num_non_empty_bins = num_covered_bins ( count >= at_least) + 

>>>                     (number of bins which have 0 < count < at_least)

>>> 

>>>Is the above definition correct?

>>> 

>>>In that case, if we define coverage as follows (as you mentioned):

>>> 

>>> 

>>>>>     num_covered_bins / num_non_empty_bins

>>> 

>>> 

>>>We are completely removing the number of empty bins from the 

>>>denominator.

>>> 

>>>Shouldn't the denominator be the total number of bins defined for a 

>>>coverpoint which is:

>>>num_all_defined_bins = num_non_empty_bins + num_empty_bins

>>> 

>>>Note that num_all_defined_bins will not include ignore/illegal bins.

>>> 

>>>Thx,

>>>Swapnajit.

>>> 

>>>Gordon Vreugdenhil wrote:

>>> 

>>> 

>>>>Arturo Salz wrote:

>>>> 

>>>> 

>>>>>Gord,

>>>>> 

>>>>>I agree that it's best to not consider empty bins and compute 

>>>>>coverage

>>>>>as:

>>>>>     num_covered_bins / num_non_empty_bins While both mechanisms 

>>>>>described by you allow coverage to be 100%, this one also exhibits 

>>>>>the nice property of being 0 when nothing is run. I believe it is 

>>>>>counterintuitive to have non-zero coverage when no bins are
covered.

>>>>> 

>>>>>     Arturo

>>>> 

>>>>Arturo,

>>>> 

>>>>That's fine with me.  An edge case -- 18.10.1 doesn't define

>> 

>>any rules

>> 

>>>>for results when the number of non-empty bins is 0.  So, should this


>>>>be defined to be:

>>>>  1) illegal

>>>>  2) 100% coverage

>>>>  3) 0% coverage

>>>> 

>>>>I don't think I would want (1) to be an LRM requirement (an 

>>>>implementation might warn of course), but for parameterized 

>>>>ignore_bins, etc. the situation might actually be reasonable.

>>>> 

>>>>If the user anticipates the scenario, I think that (2) is the best 

>>>>choice since otherwise full coverage can't be reached.

>>>> 

>>>>Gord.

>>>> 

>>>> 

>>>> 

>>>>>-----Original Message-----

>>>>>From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org] On Behalf Of


>>>>>Gordon Vreugdenhil

>>>>>Sent: Monday, January 09, 2006 12:52 PM

>>>>>To: SV_EC List

>>>>>Subject: [sv-ec] Completing my coverage thought...

>>>>> 

>>>>>Just to complete my thought regarding coverage before the call was 

>>>>>dropped -- my basic question is whether the coverage percentage is:

>>>>>    num_covered_bins / num_non_empty_bins

>>>>>or:

>>>>>    (num_covered_bins + num_empty_bins) / (num_non_empty_bins +

>>>>>num_empty_bins)

>>>>> 

>>>>>(i.e. is an empty bin not considered at all or is it trivially

>>>>>covered?)

>>>>> 

>>>>>I think that the former is perferable, but I don't know for sure.

>>>>> 

>>>>>Gord.

>>>> 

>>>>--

>>>>--------------------------------------------------------------------

>>>>Gordon Vreugdenhil                                503-685-0808

>>>>Model Technology (Mentor Graphics)                gordonv@model.com

>> 

>>--

>>--------------------------------------------------------------------

>>Gordon Vreugdenhil                                503-685-0808

>>Model Technology (Mentor Graphics)                gordonv@model.com

>> 

> 

> 

> 

 

-- 

--------------------------------------------------------------------

Gordon Vreugdenhil                                503-685-0808

Model Technology (Mentor Graphics)                gordonv@model.com
Received on Wed Jan 18 13:15:20 2006

This archive was generated by hypermail 2.1.8 : Wed Jan 18 2006 - 13:18:11 PST