This document is now published as GFD-I.077.  Thanks to the authors/editors for their efforts.

Response to public comments: the following notes indicate the stept taken since the document was returned to the authors subsequent to public comment.



#1 Steve Crumb, 2006-03-30

Issue:

I wish to encourage those responsible for carrying out these tests that
a watchful eye is kept on process capture and improvement, so as to
serve the broader community as a proposed best practice for future
interoperability testing. I also agree with others' comments that this
document should be more carefully reviewed by and agreed to by all
Standards ADs as an appropriate methodology for testing the DAIS specs.

Response: 

We hope that the document provides some clear principles that will be of
help to others developing interoperability testing strategies; we will
seek to ensure that the follow-up community practice document relating 
the results of the interoperability tests can be easily related back to
this document.  We will also make the actual tests available at that time.

We are happy for all standards based ADs to comment on the document, but
don't want there to be lengthy delays while this happens.  As the presence
of this document was flagged to all members of the GFSG early in the public
comment period, I suggest that everyone has now had an appropriate chance 
to comment.  We are keen to proceed to implement the strategy described 
here within the lifetime of current limited project funding.

#2 Malcolm Atkinson, 2006-04-04

Issue:

The only question that occured to me, was who adjudicates whether the
expected return message is correct, as many of the query languages
hosted through DAI are not so specific that simple textual comparison
will demonstrate equivalence, e.g. numbers may be in a different form,
terms or sibling document branches may be in a different order.

Response:

Where the tests are sensitive to such problems, textual comparisons must
be avoided. In some cases it will not be possible to specify the exactly
what an operation should return, as this reflects a property of the data
resource accessed by the data service. In such a case, the response message 
should be validated against its schema, and the existence of specific values 
should be tested, regardless of order and wherever possible taking into 
account different formats etc. This is practical to implement because of 
the availability of tools such as Axis which marshal XML responses into 
Java objects which can be manipulated more easily regardless of such
problems. Changes to Section 4.1 have been made in order to state
this explicitly in the document.

#3 Dave Berry, 2006-04-05

Issue:

It seems implicit in this document that complete interoperability
testing of the specification is not practical. I think it would be
useful to state this explicitly in the introduction. At present the
document seems to skirt around the issue.

Response:

The following text has been added at end of Section 1 to address
this comment: "The DAIS specifications define behaviors that may vary 
between different valid implementations of the specifications; such
differences principally reflect variety in the data resources to 
which the services provide access.  It is therefore not practical
to test every behavior that could possibly be exhibited by a DAIS 
service without implementing or emulating large numbers of data 
resource management systems. However, the testing process detailed in 
this document aims to test all mandatory features of the specifications 
and as many optional features as is practical given the capabilities
of the resource management systems accessed by the services that are
bring tested."

#4 Dave Berry, 2006-04-05

Issue:

The document suggests that WSRF properties should not be tested. This
seems odd to me. While the base WSRF functionality will no doubt be
tested elsewhere, shouldn't the DAIS tests check that an implementation
is returning the correct properties?

Response:

The WS-DAI specifications allow properties to be accessed in two ways:
through mandatory operations defined by the WS-DAI specification, and 
optionally via WS-Resource Properties. Regardless of the method used, 
access is to the same property values.  The objective of the tests is 
principally to determine whether or not property values are correct.  
Therefore we propose only to test for the correctness of property values
via the mandatory property access operations, and not to test additional
operations defined in other specifications that may be used alongside 
the DAIS specifications.

#5 Malcolm Atkinson, 2006-04-11

Issue:

I believe that the intention is to verify that correct information is
returned, e.g. for a request for ResourceProperties. I believe that all
of the functions that can be checked for consistence should be verified
and I think that was the intention of the document.

Response:

Agreed, however, we suggest that we can test for the correctness of
property values using the messages defined in the DAIS specifications
without providing large numbers of additional tests using operations
from other specifications that may not be used with DAIS.

Response to comments at GGF 17
------------------------------

#6 Stephen Pickles:

Issue:

What does single implementation tell you about interoperability of the
specifications?  Nothing - for quality assurance.

Response:

Following text added to Section 4.1:

"Tests related to features classified as optional in this document do
not impact upon the assessment of interoperability between multiple
implementations and function as a quality assurance test for the
specifications."


#7 Stephen Pickles:

Need to be careful to explain why some properties are not tested.

Response:

Added the following text to Section 4.4:

"The WS-DAI specification defines similar operations to the WSRF
specifications for property access. Regardless of the method used (the
WSRF or WS-DAI operations), access is to the same underlying properties.
It is trivial to test access to the same properties using different
operations and there is therefore no requirement to test property access
using the WSRF defined operations, as tests which access the same
properties, using the WS-DAI defined operations, will already exist in
the test suites."

#8 Stephen Pickles:

There is an interop-bof mailing list, email for comments.

Response:

Done. No comments received.	

#9 Dave Snelling:

Could test with more than one client. Ideally a cross-table of
different clients tested against different implementations should be
produced.

Response:

Section 4.2 now describes how different test clients may be used to
test implementations. 

Thanks, and sorry for the delay.  Yes, please go ahead with
those changes.  We're looking forward to the next/final
document.  Work with your ADs as needed for this revision,
then upload and assign back to gbn for the next phase.

The attached note indicates how we propose to respond to the comments made during the public comment period on the above document.  At the time of 
writing (2nd May 2006) the public comment period has completed but the status has not changed on Gridforge.  We are keen to clarify as soon as 
possible how to proceed from  here, especially as GGF17 starts in just over a week.

Norman Paton/Steve Lynden

I believe that the intention is to verify that correct information is returned, e.g. for a request for ResourceProperties. I believe that all of the 
functions that can be checked for consistence should be verified and I think that was the intention of the document.

I believe that the intention is to verify that correct information is returned, e.g. for a request for ResourceProperties. I believe that all of the 
functions that can be checked for consistence should be verified and I think that was the intention of the document.

Loaded to GGF public comment web page 4 April, 2006

I have read the document and considered its suitability as a description of the process for testing and demonstrating the validity of the three DAI 
propsed specifications. In my view it is an excellent plan.  

It is sufficient in the sense that it specifies all that is necessary for testing the extent to which these proposed specifications define the 
operations, properties and failure modes of the three specifications. 

It is also necessary, in the sense that I could not see anything that could be left out and still fully test the interoperability. In my view this is 
a very useful contribution as it demonstrates excellent preparation for interoperability testing. 

The only question that occured to me, was who adjudicates whether the expected return message is correct, as many of the query languages hosted 
through DAI are not so specific that simple textual comparison will demonstrate equivalence, e.g. numbers may be in a different form, terms or sibling
 document branches may be in a different order.

As such, it is a model that could be used by other specifications that have a similar topology between client and service.  It is particularly 
relevant for stateful services.

Malcolm

Thanks for this submission.  It looks fine, and
is now advanced to 30-day public comment.

Authors are encouraged to solicit public comments
from WG members and others.  The interop process
is an important one for the GGF, and this looks like
it will serve a valuable role.

Dear Author or Draft Editor-

Thank you for submitting your draft to the GGF Editor!  In most cases the draft 
will be assigned to an Area Director for review as the first step in the document 
publication process.

The URL provided with this note allows you to track your submission and to 
communicate with the GGF Editor or Area Director(s) who are reviewing the 
draft.  At the very bottom of the tracker display (web page) a detailed log of all 
actions related to this submission is available.

You may also wish to alert your colleagues that the draft has been submitted, and from their GridForge accounts they may wish to monitor this item in 
order to receive email any time action is taken regarding this submission.

Informational and Experimental drafts are reviewed by the GGF Editor and one 
or more Area Directors.  The GGF Editor will determine whether the draft is 
ready for public comment, or if there are items for the author(s) to address 
prior to public comment.  This initial review generally takes no more than two 
weeks.

Community Practice and Recommendations Track drafts require a GFSG review 
prior to public comment.  This can add 2-3 weeks to the initial GGF Editor 
review depending on current workload of the GFSG.

Please do not hesitate to inquire about the status of your submission at any time by way of comments added to this tracker item, which will be 
automatically emailed to the individual who appears in the "assigned to" field.

Thanks very much- 
GGF Editor