Comments from Stephan Shultz of ETSI
---
Hi all,

as promised in the last TC GRID Meeting a PTCC review of the OGF test specifications GridRPC and Byte IO. I would 
appreciate it if someone from OGF, e.g., Steve Pickles, could forward this to the relevant people in OGF.
Thanks


Firstly I see a different use of terminology.  Both documents talk about "test cases" but understand it each differently
 and also differently from PTCC point of view.

  - GridRPC is giving high level descriptions essentially of what to test.  PTCC calls that test purposes.  What GridRPC
 calls "test code" comes closer to our idea of a test case. Something which is executable .. a test script if you will.

  - The ByteIO specification is much more detailed than a test purpose, e.g., it gives detailed message contents, 
arrival of the verdict is convert in great detail, however as such the document is not executable. In our methodology we
 call this form of specification test descriptions (which is an optional intermediate step between test purpose and test
 case implementation).

In ETSI 3GPP test specifications this intermediate step has also be used. From PTCC point of view it would be nice to 
have a consistent use of terminology in specification.  Of course it would be nice to follow ETSI methodology but at the
 end that is of course up to you.

In the same way we feel it is beneficial to separate WHAT is to be tested (=purpose) and HOW to test (= description).

In the Byte IO case it these two issues seem to be mixed and they should be untangled. Clear separation of WHAT and HOW[
StS]   gives you a better idea, e.g., to get a clear idea on the passing criteria and about your actual test coverage of
 your tests . Another benefit of test purposes is that they can be understood by a much wider audience as test 
description since they do not require deep knowledge, e.g., of  protocol syntax etc.


Both documents position themselves as being a basis for interoperability testing. From PTCC point of view both are 
really specifying conformance tests. I am not sure why the term conformance is avoided.  Or maybe there is just a mix-up
 of words here which is (achievement of) interoperability versus interoperability testing.

  - GridRPC states for example that "As such, the requirement is only to test client interoperability: that multiple 
implementations of a specification provide consistent results to a suite of tests". In my opinion this just falls short 
of saying that these implementations conform to the specification which has been selected to be the basis of these so 
called interoperability tests, i.e., GFD-R.52.

  - ByteIO test case descriptions specify validation criteria at the level of bytes, i.e., WSDL messages.

In both specs it seems to me that the system under test consists of one entity, i.e., the server, being tested in 
isolation. I could be wrong.  In ETSI (formal) interoperability testing is defined as operating different 
implementations of different entities against each other, e.g., a NIFT client implementation against a GridSolve server.
 Therefore an interoperability test would describe how to stimulate and observe client as well as server implementations
 not the interface betweeen them.


The two documents are specifying tests at different levels

  - GridRPC more at what I would consider software module testing and ByteIO more as protocol/Unit testing. From our 
point of view that is fine. However I wonder in the case of GridRPC why this specification is limited to C 
implementations only. It seems there is a lot of Grid middle ware based on Java out there. Should this test 
specification not be more implementation agnostic?

  - On the ByteIO side I wonder why test purposes have to be bound so specifically to one specific scenario or initial 
configuration? Again it should be thought about to adopt true test purposes which are more abstract and leave the 
details to the test description and implementation. One...