Project Task #9041

Project WP #665: WP7 - Supporting Blue Environment: VREs Development [Months: 7-30]

Project Task #666: T7.1 Aquaculture Atlas Generation VRE [Months: 7-30]

Project Task #8830: developments of the online viewing and editing tools

Project Task #8842: Data Validation (by VRE super user):

Develop Data Validation R process

Added by Emmanuel Blondel over 2 years ago. Updated over 2 years ago.

Status:ClosedStart date:Jun 26, 2017
Priority:HighDue date:
Assignee:Emmanuel Blondel% Done:

100%

Sprint:WP07
Lead beneficiary:12 - CLS Participants:5 - FAO
Milestones:
Duration:

Description

R process will:
• rely on WFS for grabing validated feature (an aquaculture farm or set of cages for the farm, to be clarified with CLS @nlongepe@cls.fr )
• upload the feature as shapefile in GeoServer public & publish layer based on it
• generate & publish ISO 19115/19139 metadata to Geonetwork

To be plugged as SAI/Dataminer algorithm


Related issues

Related to BlueBRIDGE - Project Task #9040: AquacultureAtlasGeneration (AAPS) - Enable SAI/DataMiner/... Closed Jun 26, 2017 Jun 30, 2017
Related to BlueBRIDGE - Support #9242: Need of 2 user accounts in Geonetwork for AAPS and PAIM Closed Jul 11, 2017
Related to BlueBRIDGE - Project Task #9248: Add simple indicator of farms validation coverage by coun... Closed Jul 11, 2017
Related to BlueBRIDGE - Project Task #9252: Install geonapi package from Github on DM Closed Jul 12, 2017
Blocks BlueBRIDGE - Project Task #9042: Implement data validation from AAPS viewer Closed Jun 26, 2017
Blocked by BlueBRIDGE - Project Task #9251: Fix prod GN configuration (xml.metadata.get, mapviewer) Closed Jul 12, 2017

History

#1 Updated by Emmanuel Blondel over 2 years ago

#2 Updated by Emmanuel Blondel over 2 years ago

  • Related to Project Task #9040: AquacultureAtlasGeneration (AAPS) - Enable SAI/DataMiner/RStudio stack on VRE added

#3 Updated by Emmanuel Blondel over 2 years ago

@fabio.sinibaldi@isti.cnr.it which prod Geonetwork can I use here? There is no instance for the aquaculture atlas (I didn't request it because Lino pointed out limited hardware resources). Can I use (at least to start) this geonetwork: http://geonetwork.d4science.org/geonetwork ? If yes, what are the credentials?

#4 Updated by Fabio Sinibaldi over 2 years ago

Hi @emmanuel.blondel@fao.org, to know what resources are assigned to a certain scope, you can always use our gui here : https://services.d4science.org/infrastructure-monitor

I'm also producing a wiki documentation on SDI-Service available at https://gcube.wiki.gcube-system.org/gcube/Interaction_with_SDI-Service . It's not complete (I'm gonna publish it on the portal as soon as it's ready), but the "SDI" interface described there should be enough for you to gather GeoNetwork credentials.

#5 Updated by Emmanuel Blondel over 2 years ago

  • Related to Support #9242: Need of 2 user accounts in Geonetwork for AAPS and PAIM added

#6 Updated by Emmanuel Blondel over 2 years ago

  • % Done changed from 0 to 60
  • Participants 5 - FAO added

The data publication part of the R script is ready and functional. It includes the following workflow:
* grab Farm data from WFS
* push it as shapefile in public workspace
* create/publish the layer if not existing, otherwise update featureType (for bbox update, in case)
* create/publish the cluster layer if not existing, create it by pushing the individual farm shapefile (as separate one), otherwise, grab all features other than target individual farm: if empty collection, create it from individual farm, otherwise append new farm. The initial approach envisaged for having the cluster layer was to rely on GeoServer aggregates or "layergroups", but the latter doesn't support WFS (since grouped layers have potentially heterogenous schemas), so i've switched to a shapefile merging & update process.

Next is:
* to append metadata creation & publication/update for both individual farm layer & clustered layer
* to test data validation with incoming data from Greece (with Greek characters), to see if similar hack as for #9059 should be added (to make Geoserver treating well encoding)
* to deploy the script as DM service

#7 Updated by Emmanuel Blondel over 2 years ago

  • Related to Project Task #9248: Add simple indicator of farms validation coverage by country in AAPS viewer added

#8 Updated by Emmanuel Blondel over 2 years ago

  • Status changed from New to In Progress

R script now enriched with metadata flow

#10 Updated by Emmanuel Blondel over 2 years ago

@anton.ellenbroek@fao.org and @nlongepe@cls.fr See here http://geonetwork.d4science.org/geonetwork/srv/en/main.home some metadata how they are created/published with the data validation process. In the search type "AAPS". ASAP the script will be deployed as service.

There is room to improve the metadata from content viewpoint but also legal aspects (Anton, yet other material for people working on legal metadata elements). For content, i discussed with CNR in order to be able to fetch user profile information, see #9253 for more information. I've put in place a Lineage information where process steps are described (i copy @julien.barde@ird.fr as well as he might be interested to get inspiration from this lineage stuff), here as well we may inherit information. e.g. for the data production, we could link to another metadata to be associated with staging products.

#11 Updated by Emmanuel Blondel over 2 years ago

#12 Updated by Emmanuel Blondel over 2 years ago

  • Start date set to Jun 26, 2017

due to changes in a related task

#13 Updated by Emmanuel Blondel over 2 years ago

  • Related to Project Task #9251: Fix prod GN configuration (xml.metadata.get, mapviewer) added

#14 Updated by Emmanuel Blondel over 2 years ago

  • % Done changed from 0 to 90

Process deployed here: https://i-marine.d4science.org/group/rprototypinglab/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.AAPS_PUBLIC_PUBLISHER

The process is not fully operational, working the first time, then fails due to issue highlighted in #9251 (impossible to perform metadata GET/UPDATE). Once #9251 is solved, i will perform final testing in link to WFS-T edition with Greek farm names to tackle possible encoding issues.

#15 Updated by Emmanuel Blondel over 2 years ago

  • Related to deleted (Project Task #9251: Fix prod GN configuration (xml.metadata.get, mapviewer))

#16 Updated by Emmanuel Blondel over 2 years ago

  • Blocked by Project Task #9251: Fix prod GN configuration (xml.metadata.get, mapviewer) added

#17 Updated by Emmanuel Blondel over 2 years ago

  • % Done changed from 90 to 100
  • Status changed from In Progress to Closed

Integration tests with specific WFS-T edits (Greek far names) will be handled in separate task.
Service is operational and invokable in the AAPS viewer

#18 Updated by Emmanuel Blondel over 2 years ago

  • Status changed from Closed to In Progress

Integration test with Greek names. Same workaround as for NASO maps to be applied to conserve datastore proper UTF-8 encoding in Geoserver.

#19 Updated by Emmanuel Blondel over 2 years ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF