RE: [sv-ec] Completing my coverage thought...

From: Swapnajit Chakraborti <swapnaj_at_.....>
Date: Mon Jan 30 2006 - 05:22:19 PST
Hi Arturo,
 
Yes, I agree with you on the semantics of ignore_bins as discussed in
this mail. Also, thanks for the information on the next steps.
 
Thx,
Swapnajit.


________________________________

	From: Arturo Salz [mailto:Arturo.Salz@synopsys.com] 
	Sent: Saturday, January 21, 2006 12:35 AM
	To: Swapnajit Chakraborti; Arturo Salz; Gordon Vreugdenhil
	Cc: sv-ec@eda.org
	Subject: RE: [sv-ec] Completing my coverage thought...
	
	

	Hi Swapnajit,

	 

	So I believe we are largely in agreement.

	 

	As for the schedule. First, the committee must vote on this
issue. Once it's decided, we can add it to the list of issues to be
published as a corrigendum (for the details see
http://standards.ieee.org/faqs/Amendments_Corrigenda.ppt ). Now that the
1800-WG has voted on how to proceed next, perhaps we can discuss our
options in the technical committee.

	 

	            Arturo

	 

	
________________________________


	From: Swapnajit Chakraborti [mailto:swapnaj@cadence.com] 
	Sent: Friday, January 20, 2006 6:15 AM
	To: Arturo Salz; Gordon Vreugdenhil
	Cc: sv-ec@eda.org
	Subject: RE: [sv-ec] Completing my coverage thought...

	 

	Hi Arturo,

	 

	Thanks for your elaborate explanation. Here are my inputs on
this topic: 

	 

	- "empty" bins should be eliminated from coverage calculation (
from both numerator 

	  and denominator). Note that we need to remove them from
denominator while

	  calculating covered percentage and we need to remove them from
numerator

	  while calculating uncovered precentage. Such bins need not be
reported as well 

	  by any reporting tool as uncovered. One argument for this can
be that if user has

	  specified some bins as "ignore_bins", there is no point in
reporting them as

	  uncovered.

	- For the corner case i.e. "empty" coverpoint, option (4) looks
better than the other

	  three. But in that case, reporting tools will have to decide
how to represent this

	  to the user.

	 

	Having said that, my question is when are we going to freeze
this new semantic?

	 

	Thx,

	Swapnajit.

	 

	  

		
________________________________


		From: Arturo Salz [mailto:Arturo.Salz@synopsys.com] 
		Sent: Thursday, January 19, 2006 2:45 AM
		To: Gordon Vreugdenhil; Swapnajit Chakraborti
		Cc: sv-ec@eda.org
		Subject: RE: [sv-ec] Completing my coverage thought...

		I'll try to clarify.

		 

		In my previous response to Swapnajit I was trying to
make it clear that ignore bins 

		do not affect the "overall coverage space". What I meant
(and I believe that was the 

		gist of the question) is that an "ignore bins"
specification does not affect how bins 

		are computed. Otherwise, one might consider removing the
ignored values from the 

		space and then distribute the remaining (non ignored)
values into the resulting bins.

		 

		Until the recent discussion, what we now call "empty
bins" was not considered special 

		in any way (other than the fact that it could never be
covered). We are now refining our 

		thinking to remove the contribution to coverage of these
empty bins. This new thinking 

		does indeed change the way coverage is computed from the
way I had described it in 

		my earlier response to Swapnajit.

		 

		Our latest thinking is that empty bins should not affect
coverage. Gord has identified 

		one corner case (a coverpoint with all empty bins) and
proposed a solution, but I don't 

		believe we have consensus on how to deal with it. My
thinking is that there are four 

		alternatives (the first three were proposed by Gord):

		   1) illegal

		   2) 100% coverage

		   3) 0% coverage

		   4) eliminate from coverage

		 

		The reason for the 4th option is that an empty
coverpoint can be simultaneously considered 

		as being trivially covered and not covered. In other
words, we are devising semantics to 

		resolve the mathematical indeterminacy of dividing zero
by zero, which could indeed be 0 or 1.

		Since I don't know how we could apply L'Hopital's rule
to coverage, I believe the best thing is 

		to remove it from coverage. By that I mean the
following:

		  1) An empty coverpoint (one that consists only of
empty bins) does not affect the coverage

		     score of the covergoup, that means, remove it from
consideration from both the numerator

		     and the denominator.

		  2) An empty covergroup (one that consists entirely of
empty coverpoints) does not affect

		      overall coverage score.

		A reporting tool should probably show these empty cover
points/groups differently. I think that

		empty coverpoints are equivalent to unreachable states
in state coverage (or unreachable lines

		in line coverage), and likewise, they should not modify
the coverage score.

		 

		            Arturo

		 

		-----Original Message-----
		From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org]
On Behalf Of Gordon Vreugdenhil
		Sent: Wednesday, January 18, 2006 7:18 AM
		To: Swapnajit Chakraborti
		Cc: sv-ec@eda.org
		Subject: Re: [sv-ec] Completing my coverage thought...

		 

		 

		 

		Swapnajit Chakraborti wrote:

		 

		>  

		> Gord,

		> 

		> Thanks for clarifying the context. According to you,
one scenario that

		> can result "empty" bins is the interaction of
ignore_bins with other

		> bins.

		> Can you please elaborate a little more on this? 

		> Do you mean that in the following example, bin b1 will
be interpreted as

		> 

		> an "empty" bin? 

		 

		Based on the recent discussion, the answer is yes.

		 

		 

		> coverpoint cp1 {

		>    bins b1 = {0, 1, 2};

		>    bins b2[] = {0, 3, [7:15] };

		>    ignore_bins ig1 = {0, 1, 2};

		> } 

		> 

		> If that is the case, then ignore_bins is actually
affecting the total

		> coverage

		> space. Is that the desired behavior? In this context,
I am attaching

		> below

		> an earlier discussion, where Arturo wrote to one of my
queries that

		> ignore_bins will not

		> affect the total coverage space.

		 

		Arturo, can you respond on this?  Based on what we
discussed,

		it seems that the coverage in the example that Swapnajit
had

		asked about earlier should in fact be 100%.

		 

		Gord.

		 

		 

		> 

		> Arturo Salz wrote:

		> 

		>>Swapnajit,

		>> 

		>>An ignored coverbin does not modify the coverage
space, it only

		> 

		> indicates

		> 

		>>that the specified bins not be counted.

		>>Thus, your example would do the following:

		>>    - Create 7 auto-bins (0..6).

		>>    - Bins 0, 1, and 2 never take a hit.

		>>And, if the sampled "a" takes on all values 0-7, the
coverage is 4/7

		> 

		> => 57.1%

		> 

		>>    Answer

		> 

		> 

		> And here is the relevant example referred in Arturo's
comment:

		> 

		> 

		>>  reg [2:0] a;

		>>  coverpoint a {

		>>     ignore_bins ig_bins = {0, 1, 2};

		>>  }

		> 

		> 

		> Please let me know your comments on this. I do agree
that if ignore_bins

		> does not

		> affect total coverage space, we can never reach 100%
coverage. These

		> points

		> need to be clarified.

		> 

		> The other point you mentioned that "empty" bins can
result if there are

		> more 

		> bins than values. I found this in LRM as well and
clearly there is no

		> mention

		> whether to include them in coverage calculation. But
according to me,

		> since 

		> empty bins resulted from user specification, we should
consider them for

		> coverage calculation. So, if user specifies following:

		> 

		> reg [2:0] a;

		> coverpoint a {

		>   bins b1[10] = {[0:7]};

		> }

		> 

		> And values reached during simulation are 0:7, the
coverage will be

		> reported as 8/10, considering the "empty" bins, namely
b1[8] and b1[9]

		> in the total.

		>  

		> Thx,

		> Swapnajit.

		> 

		> 

		> 

		> 

		>>-----Original Message-----

		>>From: owner-sv-ec@eda.org [mailto:owner-sv-ec@eda.org]
On Behalf Of 

		>>Gordon Vreugdenhil

		>>Sent: Tuesday, January 17, 2006 11:18 PM

		>>To: Swapnajit Chakraborti

		>>Cc: Arturo Salz; SV_EC List

		>>Subject: Re: [sv-ec] Completing my coverage thought...

		>> 

		>>Swapnajit,

		>> 

		>>The discussion we were having was a continuation of
the SV-EC 

		>>conference call which is why you misunderstood this.

		>> 

		>>By "empty bins" here we did not mean "bins that were
never covered" but

		> 

		> 

		>>rather "bins with no elements in their value set".

		>>This arises due to the interaction of ignore_bins
and/or having more 

		>>bins than values.

		>> 

		>>The LRM has not defined whether bins that have no
elements in their 

		>>value set are supposed to be included in the
computation.

		>>Arturo and I are in agreement that they should not.
As I noted in my 

		>>follow-up email, there is still edge case behavior to
discuss - the 

		>>situation in which there are no bins with values in
their value sets.

		>> 

		>>Gord.

		>> 

		>> 

		>> 

		>>Swapnajit Chakraborti wrote:

		>> 

		>> 

		>>>Hi Arturo & Gord,

		>>> 

		>>>As per LRM, coverage for a coverpoint (section
18.10.1) is

		>> 

		>>defined as:

		>> 

		>>>  num_covered_bins/num_all_defined_bins

		>>> 

		>>>Again, num_covered_bins is actually the number of
bins for which 

		>>>at_least count is reached (count >= at_least).

		>>> 

		>>>Now, in your discussion you have mentioned
"num_non_empty_bins".

		>>>I am assuming the following definition of
"num_non_empty_bins".

		>>> 

		>>>num_non_empty_bins = num_covered_bins ( count >=
at_least) + 

		>>>                     (number of bins which have 0 <
count < at_least)

		>>> 

		>>>Is the above definition correct?

		>>> 

		>>>In that case, if we define coverage as follows (as
you mentioned):

		>>> 

		>>> 

		>>>>>     num_covered_bins / num_non_empty_bins

		>>> 

		>>> 

		>>>We are completely removing the number of empty bins
from the 

		>>>denominator.

		>>> 

		>>>Shouldn't the denominator be the total number of bins
defined for a 

		>>>coverpoint which is:

		>>>num_all_defined_bins = num_non_empty_bins +
num_empty_bins

		>>> 

		>>>Note that num_all_defined_bins will not include
ignore/illegal bins.

		>>> 

		>>>Thx,

		>>>Swapnajit.

		>>> 

		>>>Gordon Vreugdenhil wrote:

		>>> 

		>>> 

		>>>>Arturo Salz wrote:

		>>>> 

		>>>> 

		>>>>>Gord,

		>>>>> 

		>>>>>I agree that it's best to not consider empty bins
and compute 

		>>>>>coverage

		>>>>>as:

		>>>>>     num_covered_bins / num_non_empty_bins While
both mechanisms 

		>>>>>described by you allow coverage to be 100%, this
one also exhibits 

		>>>>>the nice property of being 0 when nothing is run. I
believe it is 

		>>>>>counterintuitive to have non-zero coverage when no
bins are covered.

		>>>>> 

		>>>>>     Arturo

		>>>> 

		>>>>Arturo,

		>>>> 

		>>>>That's fine with me.  An edge case -- 18.10.1
doesn't define

		>> 

		>>any rules

		>> 

		>>>>for results when the number of non-empty bins is 0.
So, should this 

		>>>>be defined to be:

		>>>>  1) illegal

		>>>>  2) 100% coverage

		>>>>  3) 0% coverage

		>>>> 

		>>>>I don't think I would want (1) to be an LRM
requirement (an 

		>>>>implementation might warn of course), but for
parameterized 

		>>>>ignore_bins, etc. the situation might actually be
reasonable.

		>>>> 

		>>>>If the user anticipates the scenario, I think that
(2) is the best 

		>>>>choice since otherwise full coverage can't be
reached.

		>>>> 

		>>>>Gord.

		>>>> 

		>>>> 

		>>>> 

		>>>>>-----Original Message-----

		>>>>>From: owner-sv-ec@eda.org
[mailto:owner-sv-ec@eda.org] On Behalf Of 

		>>>>>Gordon Vreugdenhil

		>>>>>Sent: Monday, January 09, 2006 12:52 PM

		>>>>>To: SV_EC List

		>>>>>Subject: [sv-ec] Completing my coverage thought...

		>>>>> 

		>>>>>Just to complete my thought regarding coverage
before the call was 

		>>>>>dropped -- my basic question is whether the
coverage percentage is:

		>>>>>    num_covered_bins / num_non_empty_bins

		>>>>>or:

		>>>>>    (num_covered_bins + num_empty_bins) /
(num_non_empty_bins +

		>>>>>num_empty_bins)

		>>>>> 

		>>>>>(i.e. is an empty bin not considered at all or is
it trivially

		>>>>>covered?)

		>>>>> 

		>>>>>I think that the former is perferable, but I don't
know for sure.

		>>>>> 

		>>>>>Gord.

		>>>> 

		>>>>--

	
>>>>--------------------------------------------------------------------

		>>>>Gordon Vreugdenhil
503-685-0808

		>>>>Model Technology (Mentor Graphics)
gordonv@model.com

		>> 

		>>--

	
>>--------------------------------------------------------------------

		>>Gordon Vreugdenhil
503-685-0808

		>>Model Technology (Mentor Graphics)
gordonv@model.com

		>> 

		> 

		> 

		> 

		 

		-- 

	
--------------------------------------------------------------------

		Gordon Vreugdenhil
503-685-0808

		Model Technology (Mentor Graphics)
gordonv@model.com
Received on Mon Jan 30 05:22:53 2006

This archive was generated by hypermail 2.1.8 : Mon Jan 30 2006 - 05:23:05 PST