Project Activity #3189

Project WP #665: WP7 - Supporting Blue Environment: VREs Development [Months: 7-30]

Project Task #666: T7.1 Aquaculture Atlas Generation VRE [Months: 7-30]

7.1.6 Vizualisation of data (EO + aquaculture features) with EODA portal

Added by Anton Ellenbroek over 3 years ago. Updated over 1 year ago.

Status:ClosedStart date:Nov 17, 2016
Priority:HighDue date:
Assignee:Manuel Goacolou% Done:

100%

Sprint:WP07
Participants:1 - CNR, 12 - CLS, 3 - ENG, 5 - FAO
Milestones:
Duration:

Description

Vizualisation of EO data over EODA portal (Native + crop within ROI + relevant other
layer via WMS…)

db.py Magnifier (2.86 KB) Nicolas Longépé, Oct 18, 2016 11:14 AM

aaps_publish_layers_shapefile.py Magnifier (2.31 KB) Emmanuel Blondel, Feb 16, 2017 04:25 PM


Subtasks

Project Activity #5794: Publish new Geoserver workspace "aaps" in support of 'Aqu...ClosedFabio Sinibaldi

Project Activity #5828: Evaluate feasibly to publish Spatialite layer in GeoServe...ClosedManuel Goacolou

Task #6127: Deploy SG container with data transfer service to Geoserv...ClosedRoberto Cirillo

Project Activity #7444: Access of EODA portal via VREClosedNicolas Longépé

History

#1 Updated by Pasquale Pagano over 3 years ago

  • Tracker changed from Project Task to Project Activity

#2 Updated by Nicolas Longépé over 3 years ago

  • % Done changed from 0 to 10

#3 Updated by Nicolas Longépé over 3 years ago

  • Status changed from New to In Progress

#4 Updated by Pasquale Pagano over 3 years ago

  • Assignee set to Nicolas Longépé

#5 Updated by Nicolas Longépé about 3 years ago

  • % Done changed from 10 to 40

The visualisation of S1 is ready.
There is still some work to integrate the visualisation of S2.

#6 Updated by Nicolas Longépé about 3 years ago

  • File db.pyMagnifier added
  • Participants 1 - CNR, 12 - CLS, 3 - ENG added

@CNR or @ENG: would it be possible to handle the WMS for the following 2 layers:

  • the FAO database provided in https://goo.gl/gPAZ8q see FAO_fishcages_Greece.shp
  • the output generated by the AAPS : see FishCages_Greece_v2.db which is a spatiaLite DB.

FYI, please find enclosed the declaration of the tables

#7 Updated by Pasquale Pagano about 3 years ago

  • Assignee changed from Nicolas Longépé to Gianpaolo Coro

@gianpaolo.coro@isti.cnr.it, can you check if we can perform this activity? If yes, can it be assigned to either @fabio.sinibaldi@isti.cnr.it or another person that you identify?

#8 Updated by Gianpaolo Coro about 3 years ago

  • Assignee changed from Gianpaolo Coro to Nicolas Longépé
  • Status changed from In Progress to Feedback

Hi, please let me understand it better. I will transform the shapefile file you have provided into a GIS layer. This can be done using a DataMiner algorithm that is going to be released. I will point you to the imported layer.

Meanwhile, I have questions about the DB:

1- Do you need it on a separate Database machine in the infrastructure?
2- How much storage will it require in the future (how much should we stock)?
3- Is it just for test/evaluation?

At the TCOM, you said that CLS data are going to require several TB of storage. I don't know which is the relationship between these data and the attached database. Could you clarify, please?

#9 Updated by Gianpaolo Coro about 3 years ago

Meanwhile, I give you two links to your shapefile, we just imported:

WFS:

http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/ows?service=wfs&version=1.0.0&request=GetFeature&typeName=shp_f5013197bec34e95bb56e5f36411a8bd&format=json

WMS:

http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/aquamaps/wms?service=WMS&version=1.1.0&request=GetMap&layers=aquamaps:shp_f5013197bec34e95bb56e5f36411a8bd&styles=&bbox=19.9051971435547,35.4805564880371,27.8976573944092,40.970573425293&width=512&height=351&srs=EPSG:4326&format=application/openlayers

#10 Updated by Nicolas Longépé about 3 years ago

My IT collegue Manuel Goacolou would like to access this discussion.
It seems his email address mgoacolou@cls.fr is already in the database.

When trying to recover his password, he has this message "A new password will be sent to mgoacolou@cls.fr if you can correctly answer the following question."
The question is "Unknown question" ... so he cannot recover his password.

Can you please do something with his profile ?
Thanks

#11 Updated by Manuel Goacolou about 3 years ago

Hi,

I have justed tested the WMS interface. It works fine, but the "GetCapabilities" do not respond (http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/aquamaps/wms?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetCapabilities).
We need the request for our EODA web site for auto discover of layers availability.

do you plan to implement this request (normally a mandatory for WMS)

regards

Gianpaolo Coro wrote:

Meanwhile, I give you two links to your shapefile, we just imported:

WFS:

http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/ows?service=wfs&version=1.0.0&request=GetFeature&typeName=shp_f5013197bec34e95bb56e5f36411a8bd&format=json

WMS:

http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/aquamaps/wms?service=WMS&version=1.1.0&request=GetMap&layers=aquamaps:shp_f5013197bec34e95bb56e5f36411a8bd&styles=&bbox=19.9051971435547,35.4805564880371,27.8976573944092,40.970573425293&width=512&height=351&srs=EPSG:4326&format=application/openlayers

#12 Updated by Manuel Goacolou about 3 years ago

FYI, the database .db is an example of the format we will use for the Greece case. The version of the db will be updated regularly. No need for larger storage capacities or specific machine.

The large data (several TB) mentioned by Nicolas refer to other activities which are not in frame of the Greece case. Statistic analysis of metocean data at high resolution using Sentinel-1 for example. As for now, we do not plan to do it yet within the BlueBRIDGE.

Gianpaolo Coro wrote:

Hi, please let me understand it better. I will transform the shapefile file you have provided into a GIS layer. This can be done using a DataMiner algorithm that is going to be released. I will point you to the imported layer.

Meanwhile, I have questions about the DB:

1- Do you need it on a separate Database machine in the infrastructure?
2- How much storage will it require in the future (how much should we stock)?
3- Is it just for test/evaluation?

At the TCOM, you said that CLS data are going to require several TB of storage. I don't know which is the relationship between these data and the attached database. Could you clarify, please?

#13 Updated by Gianpaolo Coro about 3 years ago

Thanks Manuel for the explanation.
As for the DB, currently we do not support the SpatiaLite extension for GeoServer. Can your data be handled using Postgres tables or other formats (e.g. NetCDF, GeoTiffs etc.)? We have optimised our geospatial infrastructure on these formats in the last years.

If it is mandatory I will open a ticket to manage this.

#14 Updated by Gianpaolo Coro about 3 years ago

I also report your email answer, which was not reported in the ticket.

<<Hi,

I have justed tested the WMS interface. It works fine, but the "GetCapabilities" do not respond (http://geoserver-dev.d4science-ii.research-infrastructures.eu/geoserver/aquamaps/wms?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetCapabilities).
We need the request for our EODA web site for auto discover of layers availability.

do you plan to implement this request (normally a mandatory for WMS)

regards>>

Ours is a standard Geoserver installation, but the layers are not public. Generally, we use a GeoNetwork instance to get the catalogue of the resources and to explore the available maps. This catalogue is different depending on the Virtual Research Environment (VRE) you are accessing from. We provide a Java client library to browse this catalogue according to the VRE visibility policies. We have also a Web interface (e.g. https://i-marine.d4science.org/group/biodiversitylab/geo-visualisation) to search for the published maps in a VRE. I know this approach can be tricky to understand at the beginning, but is the way we manage complex access and publication policies for the maps. Some of the services are indeed a cloud of services, although they appear to have one single access point.

I add my colleague @fabio.sinibaldi@isti.cnr.it to this discussion, who is the manager of our geospatial infrastructure.

#15 Updated by Manuel Goacolou about 3 years ago

Hi,

Soory for the late reply,

Due to limitations with the Shapefile format (length of variable name, only one layer per shapefile, number of file per shapefile, ...) we do not plan to use this format.
We, also, not plan to use a PostgresSQL+PostGIS instance.

We think that Spatialite is the simpler way to exchange GIS data.

Gianpaolo Coro wrote:

Thanks Manuel for the explanation.
As for the DB, currently we do not support the SpatiaLite extension for GeoServer. Can your data be handled using Postgres tables or other formats (e.g. NetCDF, GeoTiffs etc.)? We have optimised our geospatial infrastructure on these formats in the last years.

If it is mandatory I will open a ticket to manage this.

#16 Updated by Manuel Goacolou about 3 years ago

Hi,

If the GetCapabilities is not available, we will use "hard coded" informations:
- list of layers
- list of styles if available
- list of SRS/CRS: at least EPSG:900913

Do you have the GetFeatureInfo's request implemented?

#17 Updated by Pasquale Pagano about 3 years ago

The infrastructure services do support Get Capability. Could you please clarification to who you are posting your question?
I discourage the use of stati information in the software. Thanks

#18 Updated by Nicolas Longépé about 3 years ago

  • Subject changed from 7.1.6 Vizualisation of EO data with EODA portal to 7.1.6 Vizualisation of data (EO + aquaculture features) with EODA portal

#19 Updated by Pasquale Pagano about 3 years ago

  • Due date set to Dec 02, 2016

due to changes in a related task

#20 Updated by Nicolas Longépé almost 3 years ago

  • Participants 5 - FAO added

Dear all,

I have modified the AAPS output format as shapefile. Event though this solution is less elegant, it is at least a way to move forward.
The shapefile is located here:
https://goo.gl/gPAZ8q

I hope it will be easier to broacast it as WMS layer.

#21 Updated by Emmanuel Blondel almost 3 years ago

Ok, please let me know if you still consider Python as language to proceed with the publication on your side.

#22 Updated by Nicolas Longépé almost 3 years ago

Yes, preferably.

#23 Updated by Emmanuel Blondel almost 3 years ago

The link you gave me is broken (i get a message from bluebridge saying "Sorry, an error has occurred on the server when retrieving item. Either the item doesn't exist anymore or you do not have the permission to access it")

Can you send me the shapefile by email? Thanks

#24 Updated by Anton Ellenbroek almost 3 years ago

Issue with the link is fixed.

#25 Updated by Emmanuel Blondel almost 3 years ago

See attached python script that will let you publish a local zipped shapefile into Geoserver. You will need to specifify your working dir. It is assumed that shapefile is published in 'aaps' Geoserver workspace, 'aaps_datastore' (directory of shapefiles).

For customizing the layer, some piece of code will have to be added (e.g. custom styles). Please let me know if you want to provide custom style handling, if yes provide me with the symbology you want to apply, i can prepare the SLD styles, and add the piece of code required.

Let me know

#26 Updated by Manuel Goacolou over 2 years ago

Hi,

Here a status of EODA portal updates for VRE layers visualisation:

  • Upgrades of EODA are in progress:

    • connection to VRE is OK.
    • adds of S2 product layers in the specific Bluebridge instance are in progress should be finished next week.
    • tests to be finished for full integration next week
  • upgrade of the production environnement of EODA planned next week.

#27 Updated by Manuel Goacolou over 2 years ago

  • Assignee changed from Nicolas Longépé to Manuel Goacolou

#28 Updated by Nicolas Longépé over 2 years ago

We try to set up the geoserver to enable at least 6 decimals for the WFS and the GeoJSON output...
Even though we modify the adequate value in the "global setting" of https://geoserver1-p-spatial.d4science.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.ToolPage, the GeoJson outputs are still with 4 decimals which are definetely not enough for our application (as 1 degree is about 100 km, 0.0001 deg is about 10 meters)

any helps ? @emmanuel.blondel@fao.org @pasquale.pagano@isti.cnr.it

#29 Updated by Emmanuel Blondel over 2 years ago

There were issues in Geoserver related to this global setting, and its application to some WFS output formats. Its support for GeoJSON should be in the geoserver version we use, but for some reason it doesn't work, i'm investigating.

#30 Updated by Nicolas Longépé over 2 years ago

as you could maybe see with the demo, we got around this issue.
The request are now simply in epsg:900913(3857) with unit in meters, instead of epsg:4326 with unit in deg.

thanks

#31 Updated by Nicolas Longépé over 1 year ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF