Project Task #9042
Project WP #665: WP7 - Supporting Blue Environment: VREs Development [Months: 7-30]
|Status:||Closed||Start date:||Jun 26, 2017|
|Assignee:||Nicolas Longépé||% Done:|
|Lead beneficiary:||12 - CLS||Participants:|
Invokation of the web-service developed in #9041. Token required.
#2 Updated by Emmanuel Blondel over 2 years ago
- Subject changed from Implement DataMiner validation web-service (Dataminer) from AAPS viewer to Implement data validation from AAPS viewer
Data validation includes:
• WFS-T edits (for updating status value)
• Invokation of validation web-service (for publishing data to public) #9041
#3 Updated by Emmanuel Blondel over 2 years ago
@firstname.lastname@example.org here is an example of Execute request for invoking the Data validation process: http://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=***your_token***&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER&DataInputs=country=Greece;farm_id=234;
Note you have to specify your token (you can grab it from the AAPS VRE, let me know in case of issues with that).
For the process, i could enrich further it to handle some minimal business validation rules, that in case of no compliance, returns an exception. A typical example of validation rule is to have non null value for the farm name. Possibly similar for other fields (species?). We may talk of that with @email@example.com and @firstname.lastname@example.org on my side the R process is easily and quickly customizable.
#4 Updated by Emmanuel Blondel over 2 years ago
- with the temporal coverage support of layers, an extra parameter 'layer_name' has been added, and needs to be specified by the client to indicate which time layer is targeted by the validation task.
- 3 business validation rules have been added in the AAPS PUBLIC PUBLISHER:
* 'name' field cannot be empty"
* 'editors' field cannot be empty"
* 'edit_time' field cannot be empty"
In these cases, the process will be a successful HTTP request (200) but the output will return the error and status = false, which means nothing will be published into public domain if one of the above business validation rules is not fulfilled. The test public layer/metadata reachable at http://geonetwork-aquacultureatlasgeneration.d4science.org/geonetwork/srv/en/main.home have been published after a test WFS-edit of the above field.
#9 Updated by Nicolas Longépé about 2 years ago
- File RLOGa20d7ab2-8db0-462f-af94-6a866c133bf2.txt added
Yes, we manage to plug the WFS-T. It works.
But for the validation, if you can try this script:
with farm_id like 3 for example, so trying to validate the farm with farm-id = 3, it seems that the system tries to update farm with ID =1
please look at the log here.
PS: enjoy your stay
#10 Updated by Emmanuel Blondel about 2 years ago
Indeed, we need to have someone from CNR looking into that. There is maybe a misalignement of input param between script and process definition, but on the other hand this issue you described is not new as we had similar several times with dataminer working with Levi in MPA.
#15 Updated by Gianpaolo Coro about 2 years ago
Hi Nicolas, the issue is that the default value in the Input/Output panel was not reflected in the code: one was set to 0 (panel) and the other to 1 (code). Perhaps one of the two values was manually edited after pressing the +Input button. I have changed the one in the code and repackaged. I cannot test, but it should work. Please, let me know.
In the SAI FAQs we have collected all these frequent issues, among which you also find the solution to this problem:
When something does not work with SAI, we usually encourage to search in these FAQs first.
#18 Updated by Emmanuel Blondel about 2 years ago
Nicolas, you miss the layer_name parameter (the example of request above doesnt specify it). Pls look at the I/O definition at: https://i-marine.d4science.org/group/aquacultureatlasgeneration/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER
Thanks Gianpaolo for your feedback
#20 Updated by Nicolas Longépé about 2 years ago
after validated the farm_id 5 for example,
the geoserver does not seem to be updated
The layer "public:grc_farms_2017-01-01_2017-12-31_farm_all" where the modification is supposedly done (@email@example.com right ??) does not concatenate with the new validated farm ...
or I am missing something else ...
Sorry for bothering you all :-)
#23 Updated by Nicolas Longépé about 2 years ago
indeed, we manage to valide farm. no pb !
The process is a bit long (more than 30 sec), so for the validation process via the web app:
- there will be a color code for the status of farms based on the information of staging data
- there is a validation button activing the validation process, but for the upcoming version of the webapp there will no handle of the feedback on the status of the validation process. Maybe we could integrate a pop-up telling the user the validation process went OK via a background handler.
#24 Updated by Emmanuel Blondel about 2 years ago
I assumed that you could handle quickly some minimalistic UI for the validation message. IMHO it is quite straightforward tand quick to implement. No need to popup just adding a span message in boostrap style that appears temporarily to inform user.
For the time processing there are several considerations to take into account. First: a workload due to DM process initialisation which could be residual for long processes but not for quick data management processes expected to be invoked in webapp clients. DM is ok for these long processes but for atomic processes initialisation can be higher tjan the actual business logic. It's the case here. The R process runs in few sec in R. In the past, we already discussed with CNRabout complementary tech to DM to fulfill these processing needs. Lighter tech isto envisage
There are considerations proper to the business logic. Validating a farm is not only switching a boolean value but a series of complex actions on the AAPS SDI including: pick up data, push it to GS, create/ publish metadata in GN, update collection-level dataset in GS/GN (each time you validate a farm it is merged to a collection of validated farms). In addition, for the latter action I had to do some workaround with GS datastore config because of a bug in GS that doesnt persist utf-8 encoding with CRUD operations. This adds a bit of workload yo the process but guarantees proper encoding of characters, eg greek characters.