Task #12856

Verification of GRSF merging process

Added by Yannis Marketakis about 1 year ago. Updated 7 months ago.

Status:NewStart date:Nov 11, 2018
Priority:HighDue date:Nov 24, 2018
Assignee:Yannis Marketakis% Done:

80%

Category:-
Sprint:GRSF
Milestones:
Duration: 10

Description

Before proceeding with the actual merging of GRSF records, it is important to verify how it will be done for each case (e.g. merging of time-series, what happens with semantic identifiers, etc.).

FAO and FORTH need to verify the alternatives and then proceed with the actual merging. FAO and FORTH will discuss these issues in a call (data to be defined).


Related issues

Blocks StocksAndFisheriesKB - Bug #12277: Merge action is not working In Progress Jul 31, 2018

History

#1 Updated by Yannis Marketakis about 1 year ago

  • Blocks Bug #12277: Merge action is not working added

#2 Updated by Yannis Marketakis about 1 year ago

  • Priority changed from Normal to High

#3 Updated by Yannis Marketakis about 1 year ago

Merging stocks

Pre-requisites

  • species in the records under merging should be the same. If not the same, one of them could be unknown. Otherwise, the merging should be forbidden. --> To check with CNR

Process

  • the records under merging (RUM) are removed and a new one is created having as sources all the legacy records of the ones under merging.
  • the GRSF semantic identifier of the merged record will contain the species and the merged value containing all the assessment areas.
  • the time series of the merged record should be a result of the timeseries of RUM. Ideally, for identifying which value comes from which source (in the case of merging between records derived from the same source), we could add the URL, or the UUID of the legacy records with the timeseries. --> to check with CNR if this is feasible
  • assessment areas: include all from RUM
  • DB Source: include all from RUM
  • GRSF type: Assessment Unit -> Marine Resource (in any case this can change from the management panel after the merging)
  • Short name: short names from RUM concatenated with ';' (in any case this can change from the management panel after the merging)
  • Similar records: include all from RUM
  • Species: no change
  • Stock name: re-construct based on the contents of the merged record
  • Data owner: include all from RUM
  • Scientific Advice: merge from RUM
  • state and trend: include all from RUM
  • spatial: FAO to decide

Merging Fisheries

Pre-requisites

ALL of the following conditions should be met to support merging fisheries

  • species in the records under merging should be the same. If not the same, one of them could be unknown. Otherwise, the merging should be forbidden.
  • flag state in the records under merging should be the same. If not the same, one of them could be unknown. Otherwise, the merging should be forbidden.
  • fishing gear in the records under merging should be the same. If not the same, one of them could be unknown. Otherwise, the merging should be forbidden.

Process

The process is the same with stocks merging

#5 Updated by Yannis Marketakis about 1 year ago

Thank you @emmanuel.blondel1@gmail.com .
I will invoke this in the merging process and let you know if we face any problems.

#6 Updated by Yannis Marketakis 7 months ago

  • % Done changed from 0 to 50

A first working implementation of merging GRSF Stock records has been implemented.
At this point, I would like some feedback/verification from @aureliano.gentile@fao.org about the results.
More specifically you will find in https://docs.google.com/spreadsheets/d/1sFTZbMhKY7FMZi7IccNYO289KAvIu-bi93oi2mRk1-8/edit?usp=sharing the creation of two new merged records (each one found n a different sheet). There you will see the properties of the original records (the ones under merging) and the final one. Please check the merged one and let me know if you find any error.

Apart from the records presented in the document, there are others that need to be updated (e.g. stock records based on similarity or fishery records that exploit the stocks under merging).

#7 Updated by Aureliano Gentile 7 months ago

provided that this ticket is for manual merge (the automatic merge is more stringent), please note that in the google spreadsheet the cell d7 should be modified as now it contains also the UNK code.

I had some exchange with @anton.ellenbroek@fao.org and @marketak@ics.forth.gr and we concluded the following implementation rules:
1) merge of records with standard area codes, the merged records concatenates all area codes,
2) merging standard with UNK, the unknown codes are ignored,
3) merging two records both with unknown area codes produce a record with undefined areas

#8 Updated by Yannis Marketakis 7 months ago

Thanks @aureliano.gentile@fao.org.
This is fixed now.

In addition, we've made some progress with the merging of GRSF fishery records.
You'll find a new tab with the result of the merging between two GRSF fisheries. In addition, I would like your feedback as regards the data owner of the merged fishery record. Currently, GRSF Fisheries accept only one value for the data owner, which causes issues when the fishery records under merging have different data owners (can something like this happen?)

#9 Updated by Yannis Marketakis 7 months ago

  • % Done changed from 50 to 80

#10 Updated by Aureliano Gentile 7 months ago

Thanks Yannis, apologies for the delayed answer but we are busy with the preparation of the FIRMS/GRSF meeting.
I am not sure about the data owner conflict, the GRSF merged record can have associated data, each datum is referenced and owned by its specific data owner, not the entire record per se. What am I missing there?

#11 Updated by Yannis Marketakis 7 months ago

Let me move one step back and ask it on a different way.

Can 2 (or more) GRSF fishery records be merged if they have different data owners?

If the answer is YES, then which data owners do we keep? Both of them or just one (by prioritizing them per source)?

Also available in: Atom PDF