As promised in the last TC GRID Meeting a PTCC review of the OGF test
specifications GridRPC and Byte IO.
I would appreciate it if someone from OGF, e.g., Steve Pickles, could
forward this to the relevant people in OGF.
Thanks
 
Firstly I see a different use of terminology. Both documents talk about
"test cases" but understand it each differently
- and also differently from PTCC point of view.
GridRPC is giving high level descriptions essentially of what to test. PTCC
calls that test purposes. What GridRPC calls "test code" comes closer to our
idea of a test case. Something which is executable .. a test script if you
will.
The ByteIO specification is much more detailed than a test purpose, e.g., it
gives detailed message contents, arrival of the verdict is convert in great
detail, however as such the document is not executable. In our methodology
we call this form of specification test descriptions (which is an optional
intermediate step between test purpose and test case implementation).
In ETSI 3GPP test specifications this intermediate step has also be used.
>From PTCC point of view it would be nice to have a consistent use of
terminology in specification.
Of course it would be nice to follow ETSI methodology but at the end that is
of course up to you.
In the same way we feel it is beneficial to separate WHAT is to be tested
(=purpose) and HOW to
test (= description). In the Byte IO case it these two issues seem to be
mixed and they should be
untangled. Clear separation of WHAT and HOW[StS]   gives you a better idea,
e.g., to get a clear
idea on the passing criteria and about your actual test coverage of your
tests . Another benefit of test purposes is that they can be understood by a
much wider audience as test description since they do not require deep
knowledge, e.g., of  protocol syntax etc.
 
Both documents position themselves as being a basis for interoperability
testing. From PTCC point of view both are really specifying conformance
tests. I am not sure why the term conformance is avoided.
Or maybe there is just a mix-up of words here which is (achievement of)
interoperability versus interoperability testing.
GridRPC states for example that "As such, the requirement is only to test
client interoperability: that multiple implementations of a specification
provide consistent results to a suite of tests". In my opinion this just
falls short of saying that these implementations conform to the
specification which has been selected to be the basis of these so called
interoperability tests, i.e., GFD-R.52.
ByteIO test case descriptions specify validation criteria at the level of
bytes, i.e., WSDL messages.
In both specs it seems to me that the system under test consists of one
entity, i.e., the server, being tested in isolation. I could be wrong.
In ETSI (formal) interoperability testing is defined as operating different
implementations of different entities against each other, e.g., a NIFT
client implementation against a GridSolve server. Therefore an
interoperability test would describe how to stimulate and observe client as
well as server implementations not the interface betweeen them.
 
The two documents are specifying tests at different levels - GridRPC more at
what I would consider software module testing and ByteIO more as
protocol/Unit testing. From our point of view that is fine. However I wonder
in the case of GridRPC why this specification is limited to C
implementations only. It seems there is a lot of Grid middle ware based on
Java out there. Should this test specification not be more implementation
agnostic?
On the ByteIO side I wonder why test purposes have to be bound so
specifically to one specific scenario or initial configuration? Again it
should be thought about to adopt true test purposes which are more abstract
and leave the details to the test description and implementation. One
approach could be to change from checking rigidly...