Project Task #9042

Project WP #665: WP7 - Supporting Blue Environment: VREs Development [Months: 7-30]

Project Task #666: T7.1 Aquaculture Atlas Generation VRE [Months: 7-30]

Project Task #8830: developments of the online viewing and editing tools

Project Task #8842: Data Validation (by VRE super user):

Implement data validation from AAPS viewer

Added by Emmanuel Blondel over 2 years ago. Updated over 1 year ago.

Status:ClosedStart date:Jun 26, 2017
Priority:UrgentDue date:
Assignee:Nicolas Longépé% Done:

90%

Sprint:WP07
Lead beneficiary:12 - CLS Participants:
Milestones:
Duration:

Description

Invokation of the web-service developed in #9041. Token required.

RLOGa20d7ab2-8db0-462f-af94-6a866c133bf2.txt Magnifier (3.45 KB) Nicolas Longépé, Oct 03, 2017 04:25 PM

RLOG8dfaa3d7-6606-4031-bdc5-6edd0f58e5d8.txt Magnifier (3.45 KB) Nicolas Longépé, Oct 05, 2017 01:53 PM


Related issues

Blocked by BlueBRIDGE - Project Task #9041: Develop Data Validation R process Closed Jun 26, 2017

History

#1 Updated by Emmanuel Blondel over 2 years ago

#2 Updated by Emmanuel Blondel over 2 years ago

  • Subject changed from Implement DataMiner validation web-service (Dataminer) from AAPS viewer to Implement data validation from AAPS viewer

Data validation includes:
• WFS-T edits (for updating status value)
• Invokation of validation web-service (for publishing data to public) #9041

#3 Updated by Emmanuel Blondel over 2 years ago

@mgoacolou@cls.fr here is an example of Execute request for invoking the Data validation process: http://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=***your_token***&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER&DataInputs=country=Greece;farm_id=234;

Note you have to specify your token (you can grab it from the AAPS VRE, let me know in case of issues with that).

For the process, i could enrich further it to handle some minimal business validation rules, that in case of no compliance, returns an exception. A typical example of validation rule is to have non null value for the farm name. Possibly similar for other fields (species?). We may talk of that with @nlongepe@cls.fr and @anton.ellenbroek@fao.org on my side the R process is easily and quickly customizable.

#4 Updated by Emmanuel Blondel over 2 years ago

Additional notes:
- with the temporal coverage support of layers, an extra parameter 'layer_name' has been added, and needs to be specified by the client to indicate which time layer is targeted by the validation task.
- 3 business validation rules have been added in the AAPS PUBLIC PUBLISHER:
* 'name' field cannot be empty"
* 'editors' field cannot be empty"
* 'edit_time' field cannot be empty"
In these cases, the process will be a successful HTTP request (200) but the output will return the error and status = false, which means nothing will be published into public domain if one of the above business validation rules is not fulfilled. The test public layer/metadata reachable at http://geonetwork-aquacultureatlasgeneration.d4science.org/geonetwork/srv/en/main.home have been published after a test WFS-edit of the above field.

#5 Updated by Nicolas Longépé about 2 years ago

  • % Done changed from 0 to 40

We are now working on the symbology so that SU will know which farms would need validation or not

#6 Updated by Nicolas Longépé about 2 years ago

@emmanuel.blondel@fao.org trying to run the validation process, it seems that the farm-id 1 is always validated. Indeed the farm_id seems to be not taken into consideration.

#7 Updated by Anton Ellenbroek about 2 years ago

In the absence of Emmanuel, can CNR handle this issue?

Thanks for informing us, Anton.

#8 Updated by Emmanuel Blondel about 2 years ago

Hi Nicolas can you further explain what's not working? farm_id should be the attribute explicitely handled through the schema. Please let us know if you succeeded in plugin the WFS-T for editing. Greetings from Tokyo

#9 Updated by Nicolas Longépé about 2 years ago

Yes, we manage to plug the WFS-T. It works.

But for the validation, if you can try this script:
http://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=***your_token***&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER&DataInputs=country=Greece;farm_id=234;

with farm_id like 3 for example, so trying to validate the farm with farm-id = 3, it seems that the system tries to update farm with ID =1
please look at the log here.

PS: enjoy your stay

#10 Updated by Emmanuel Blondel about 2 years ago

Indeed, we need to have someone from CNR looking into that. There is maybe a misalignement of input param between script and process definition, but on the other hand this issue you described is not new as we had similar several times with dataminer working with Levi in MPA.

#11 Updated by Pasquale Pagano about 2 years ago

Either @gianpaolo.coro@isti.cnr.it or @scarponi@isti.cnr.it could you look into this issue? thanks

#12 Updated by Gianpaolo Coro about 2 years ago

Hi, could you please share the WFS-T Project folder with me? What VRE are you using?

#13 Updated by Emmanuel Blondel about 2 years ago

VRE is AquacultureAtlasGeneration. I've now shared the process 'AAPS_public_publisher' with you and Paolo

#14 Updated by Nicolas Longépé about 2 years ago

  • Priority changed from Normal to Urgent
  • Assignee changed from manuel goacolou to Gianpaolo Coro

@gianpaolo.coro@isti.cnr.it @scarponi@isti.cnr.it Any feedback on this issue ?

#15 Updated by Gianpaolo Coro about 2 years ago

Hi Nicolas, the issue is that the default value in the Input/Output panel was not reflected in the code: one was set to 0 (panel) and the other to 1 (code). Perhaps one of the two values was manually edited after pressing the +Input button. I have changed the one in the code and repackaged. I cannot test, but it should work. Please, let me know.

In the SAI FAQs we have collected all these frequent issues, among which you also find the solution to this problem:

https://wiki.gcube-system.org/gcube/Statistical_Algorithms_Importer:_R_Project_FAQ#An_algorithm_does_not_receive_input_from_the_interface

When something does not work with SAI, we usually encourage to search in these FAQs first.

#16 Updated by Nicolas Longépé about 2 years ago

Thanks for your prompt feedback

Indeed, the right farm_id is now taken into account, but there is still an issue.

#17 Updated by Gianpaolo Coro about 2 years ago

That seems a internal process issue of missing namespace "XPath error : Undefined namespace prefix" . I can't help here.

#18 Updated by Emmanuel Blondel about 2 years ago

Nicolas, you miss the layer_name parameter (the example of request above doesnt specify it). Pls look at the I/O definition at: https://i-marine.d4science.org/group/aquacultureatlasgeneration/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER
Thanks Gianpaolo for your feedback

#19 Updated by Emmanuel Blondel about 2 years ago

  • Assignee changed from Gianpaolo Coro to Nicolas Longépé

#20 Updated by Nicolas Longépé about 2 years ago

Thanks !

after validated the farm_id 5 for example,
the geoserver does not seem to be updated
http://geoserver-aquacultureatlas.d4science.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage?1

The layer "public:grc_farms_2017-01-01_2017-12-31_farm_all" where the modification is supposedly done (@emmanuel.blondel@fao.org right ??) does not concatenate with the new validated farm ...

or I am missing something else ...

Sorry for bothering you all :-)

#21 Updated by Emmanuel Blondel about 2 years ago

please check the output of the process and paste it here. i'm not referringto tje xml response of the service but the json output file referenced within the service out xml. this json will tell you if there as some error and what.

#22 Updated by Emmanuel Blondel about 2 years ago

@nlongepe@cls.fr where do you stand on this?

#23 Updated by Nicolas Longépé about 2 years ago

indeed, we manage to valide farm. no pb !

The process is a bit long (more than 30 sec), so for the validation process via the web app:
- there will be a color code for the status of farms based on the information of staging data
- there is a validation button activing the validation process, but for the upcoming version of the webapp there will no handle of the feedback on the status of the validation process. Maybe we could integrate a pop-up telling the user the validation process went OK via a background handler.

#24 Updated by Emmanuel Blondel about 2 years ago

I assumed that you could handle quickly some minimalistic UI for the validation message. IMHO it is quite straightforward tand quick to implement. No need to popup just adding a span message in boostrap style that appears temporarily to inform user.

For the time processing there are several considerations to take into account. First: a workload due to DM process initialisation which could be residual for long processes but not for quick data management processes expected to be invoked in webapp clients. DM is ok for these long processes but for atomic processes initialisation can be higher tjan the actual business logic. It's the case here. The R process runs in few sec in R. In the past, we already discussed with CNRabout complementary tech to DM to fulfill these processing needs. Lighter tech isto envisage

There are considerations proper to the business logic. Validating a farm is not only switching a boolean value but a series of complex actions on the AAPS SDI including: pick up data, push it to GS, create/ publish metadata in GN, update collection-level dataset in GS/GN (each time you validate a farm it is merged to a collection of validated farms). In addition, for the latter action I had to do some workaround with GS datastore config because of a bug in GS that doesnt persist utf-8 encoding with CRUD operations. This adds a bit of workload yo the process but guarantees proper encoding of characters, eg greek characters.

#25 Updated by Nicolas Longépé about 2 years ago

  • % Done changed from 40 to 90
  • Status changed from New to In Progress

Activity almost finished ...
Todo: UI for validation message

#26 Updated by Nicolas Longépé over 1 year ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF