FAO Tuna Atlas VRE - Additions

The FAO Tuna Atlas is accessible here: https://i-marine.d4science.org/web/fao_tunaatlas

Main goal

  • Overall description: FAO Tuna Atlas will be the reference VRE for Fisheries (gridded or tabular) data collection, harmonization and standardization, analysis and dissemination. It is already available, but it needs further additions based on user requirements. These are related to:

    • TabMan improvements,
    • StatMan extensions,
    • SpeciesDiscovery integration (load presence and absence records and harmonize).
    • Together with IRD, FAO plans to develop a reporting module (IRD suggest to use knitr, sharelatex, R-Shiny) for the definition and design of persistent reports (one one or more data tables). The reports will be a mix of text and graphs / figures to represent a true stock analysis (not based on a algorithm, but capturing expert opinion)

Big Picture

The Figure below illustrates the workflow to collate datasets from the different tuna RFMOs, turn the data into standardized format which can be loaded into a spatialized datawarehouse and used to generate a set of services and indicators

tuna_atlas_workflow.png

This FAO Tuna Atlas VRE will initially provide the base from which to develop additional features. Future features are likely to be added to support:

  1. Data management services: metadata and data formats, access protocols, compliance with RDA working group recommendations, OGC standards, SDMX,
  2. Algorithms and models to extract and visualize data,
  3. training and learning materials in support to stock assessment teams.
  • The audience

    • developers; enrich the features of Tabman (ENG), add functions and algorithms to StatMan (FAO - Starting January).
    • individual researchers; load and harmonize tabular data, run StatMan algorithms/models.
    • data management teams; once a relevant set of algorithms is identified (pending action for FAO), the teams that need the data can be identified. This will be similar to the BFT assessment team of IRD (reported in that VRE)
    • external applications; In this VRE, no external applications to integrate with have been identified.
    • general public; find reports of stock assessment (model output) in human readable formats; this may be a pdf or html report, or graphs and other static formats. The core metadata of each product will contain a clear reference to owner, process, datasets, citation and copyright.
  • User roles needed

    • data importer,
    • data manager,
    • data analyst/specialist,
    • data dissemination
  • Data

    • data input;
    • data management
    • data out
  • Estimations

    • The estimated effort for FAO to support the development of the technologies is 4 PM, starting in January with 40% time allocation.
  • Tickets capturing the activity

In case the VRE will also integrate existing community software, rely on integrated services, or support existing software (with data, services, endpoints, interactive features), please list:

  • RStudio: IDE for R to be set up in the VRE (#4217)
  • Geonetwork: metadata catalog (#7896), Java, Open Source. Related tickets: #7455
  • Geoserver: spatial data server (#7451, #8674), Java, Open Source
  • Geometa: R Github repository (Emmanuel Blondel, FA0) to create OGC compliant metadata, Open Source. Related tickets: #7455

Workplan

  • T0 + 6 months: the first period has been dedicated to load and execute codes and data in the infrastructure: ** Deployment of RStudio in the infrastructure ** ETL R codes (Extract, Transform, Load) and database
  • Step 2 (T+6, T+15):

    • SQL codes: spatial datawarehouse deployment in the infrastructure (#4217),
    • ongoing work has been presented to partners to collect feedback: ICCAT, IOTC, IATTC, Google
    • a first set of R codes have been deployed with Dataminer to get a WPS access out of the infrastructure (#4511, #5544)
    • embedding R codes with OpenCPU / WPS to give access to the services out of the VRE environment
  • Step 3 (T+15 -> now):

    • the VRE integrates new software components: Geonetwork instance for metadata (#7896), Geoserver spatial data server (#7451), Thredds (#1983 ,#6760),
    • work on new set of R codes (created upon R codes provided by FAO, E. Blondel: https://github.com/eblondel/geometa/) to populate Geonetwork and Geoserver with proper metadata and datasets (to be made accessible with WMS, WFS, WCS access protocols)
    • integration of WPS / OpenCPU / shiny services within the VRE by using portlets (#8272,#5823),
    • the main datasets of the VRE have proper metadata and access protocols (WMS, WFS, WCS)
    • the main datasets of the VRE are delivered with widely used data formats (CSV, NetCDF) and comply with RDA recommandations for Gridded Fisheries data.
    • only metadata relevant for this VRE are displayed in the dedicated CKAN catalog of the VRE (#4281)
    • the users will be able to create dynamic reports by using Sharelatex within the VRE. Requires first sharelatex to work properly: #1825, #3153

References & presentations

Related references :

  • Taconnet Paul, Chassot Emmanuel, Guitton J., Fiorellato F., Anello E., Barde Julien. Data toolbox for fisheries : the case of tuna fisheries. Victoria Mahé : CTOI, 2016, (IOTC-2016-WPDCS12-27-Rev1), 23 p. multigr.. Working Party on Data Collection and Statistics (WPDCS), 12., Victoria (SEY), 2016/11/28-30.

Related presentations :

List of IRD algorithms

Since the beginning of the projetc, IRD is working on and deploying a large set of algorithms. Open source codes are stored and can be compilet on the infrastructure. Some of them have been deployed to make them available as services to process tuna datasets (transformation, storage, visualization).

Number VRE / use case WPS Identifier of the algorithm Environment (Development RProtolab or Production) Goal / Summary Keywords Code URL (Workspace or
github)
Author / Contact OPENCPU Web page Output example
1 FAO Tuna Atlas VRE TUNA_ATLAS_DATA_ACCESS Development This R code enables users to adapt a SQL query to get data from Sardara database storing global Eastern Bluefin Tuna, BFT-E, Stock Assessment, VPA, FLR, netCDF deprecated julien.barde@ird.fr http://mdst-macroes.ird.fr/tmp/TUNA_ATLAS_DATA_ACCESS.html
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
2 FAO Tuna Atlas VRE CATCHES_AGGREGATED_FOLLOWING_A_SELECT_VARIABLE Development Catches Aggregated Following A Select Variable
The outputs are temporal and spatial distribution of the catches aggregated following a selected variable and given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches, Effort https://goo.gl/zi66gc paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_selected_dimension.html https://goo.gl/D2QYmi
3 FAO Tuna Atlas VRE CATCHES_BY_FLAGS Development Catches By Flags
The output is a plot of the catches by flags given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Flag https://goo.gl/qHTS6e paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_flags.html
4 FAO Tuna Atlas VRE CATCHES_BY_FLAGS_SIMPLIFIED_VERSION Development Catches By Flags Simplified Version
The output is a plot of the catches by flags given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Flag https://goo.gl/fd7NVL paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_flags_simplified_version.html
5 FAO Tuna Atlas VRE CATCHES_BY_GEAR_SIMPLIFIED_VERSION Development Catches By Gear Simplified Version
The output is a plot of the catches by gear given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species https://goo.gl/dOByC1 paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_gear_simplified_version.html
6 FAO Tuna Atlas VRE CATCHES_BY_GEARS Development Catches By Gears
The output is a plot of the catches by gears for tuna fisheries given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches, Fishing gears https://goo.gl/cZstNJ paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_gear.html https://goo.gl/Qbu7Xz
7 FAO Tuna Atlas VRE CATCHES_BY_SPECIES Development Catches By Species
The output is a plot of the catches by species given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species https://goo.gl/tPwG8W paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_species.html
8 FAO Tuna Atlas VRE CATCHES_BY_SPECIES_SIMPLIFIED_VERSION Development Catches By Species Simplified Version
The output is a plot of the catches by species given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species https://goo.gl/lKbjvb paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_species_simplified_version.html
9 FAO Tuna Atlas VRE CATCHES_BY_TYPE_OF_SCHOOL Development Catches By Type Of School
The output is a plot of the catches by type of school given the filters applied by the user
Compute Fisheries Indicators From Own Formatted Dataset
Compute some fisheries indicators (plots and maps) from a dataset that you have previously formatted and imported through the algorithm Import Fisheries Form...
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches, Shool Type https://goo.gl/gT2jtn paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_by_schooltype.html
10 FAO Tuna Atlas VRE GLOBAL_CATCHES Development Global Catches
The output is a plot of the catches given the filters applied by the user
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Global Catches https://goo.gl/zi66gc paul.taconet@ird.fr http://mdst-macroes.ird.fr/BlueBridge/Tuna_Atlas/catches_global.html
11 FAO Tuna Atlas VRE TunaAtlasToNetCDF
TO BE DEPLOYED Generic script R to Transform Tuna Atlas data (stored in SARDARA) to NetCDF Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Global Catches https://goo.gl/lewZpf taha.imzilen@ird.fr DEPLOYED http://mdst-macroes.ird.fr:8080/thredds/dodsC/BlueBridgeCatalog/fao_tunaatlas_VRE/SARDARA_catches_species-gear_5deg_yearly_2010-12-31_2014-12-31.ncml
12 FAO Tuna Atlas VRE CodeListMapping
TO BE DEPLOYED Map input dataset with code lists (i.e. merge):

* InputDF: A source dataset (data.frame)
* DimensionsToMap: The dimensions to map. Name of the columns in the source dataset
* MappingVectors: A list of mapping vectors. Same order than the DimensionsToMap
* RuleNonMappedValues: A rule to apply for the data that have not been mapped
* MappedDimensions: A list of boolean that tell wether the original mapped dimensions are kept or not
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species paul.taconet@ird.fr TO BE DEPLOYED
13 FAO Tuna Atlas VRE GroupRedundantValues TO BE DEPLOYED Group into one row the strata that are redundant (i.e. the strata that have the same dimensions):
* InputDF: A source dataset (data.frame)
* RuleNonEqualFactValue: the rule to apply if the strata have same dimensions but different values of fact ({Mean, ?, ?}). We could have a threshold saying ""if the values are different less than the threshold (eg 25%) we make the average of the values. else ???
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species paul.taconet@ird.fr TO BE DEPLOYED
14 FAO Tuna Atlas VRE BestStrata_FactUnit
TO BE DEPLOYED In the input dataset, the same strata might be expressed in several units. For instance, a strata might be expressed in weight of fishes but also in number of fishes catches. This function allows to remove the redundant strata, based on the choices of units to keep and units to drop:
* InputDF: A source dataset (data.frame)
* DimensionsToCompare: the dimensions to take into account for the comparison of the strata.
* ValuesUnitsToConserve: The units to conserve (vector of strings)
* ValuesUnitsToDrop: The units to drop, ie if a strata with UnitsToDrop also exists with UnitsToConserve, this strata will be removed
* DropUnitsToDropUniqueStrata: a boolean telling the rule for the strata that only exist in UnitsToDrop (keep these strata or remove them)
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species paul.taconet@ird.fr TO BE DEPLOYED
15 FAO Tuna Atlas VRE BestStrata_MoreDisaggregatedInfo
TO BE DEPLOYED In the input dataset, there might be absence of stratification for some dimensions in some strata. For instance, a strata might be expressed "All flags included", or "All types of school included". These same strata might also be stratified, ie we might have the stratification of the strata by flags, or by type of school. This function compares the strata according to a selected dimension, and for the strata where both stratification and absence of stratification is available (eg "All included" and stratification), it removes the row "All included". This way, only the more exhaustive information (ie more disaggregated) is kept:

* InputDF: A source dataset (data.frame)
* Dimension: the name of the dimension to take into account
* DimensionAggregatedValue: In the Dimension, the values for "all included". All the other values will be considered as disaggregated values for the Dimension.
* RuleNonEqualFactValue: the rule to apply if the sum of the disaggregated strata is not equal to the aggregated strata ({StillRemoveAggregated, ?}) . We could have a threshold saying "if the values are different less
than the threshold (eg 25%) we make the average of the values. else ???"
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species paul.taconet@ird.fr TO BE DEPLOYED
16 FAO Tuna Atlas VRE UnitsConversion TO BE DEPLOYED Converts fact values according to a dataset of factors of conversion:

* InputDF: a source dataset (data.frame)
* ConversionFactorsDF: a dataset of conversion factors (data.frame)
* Other arguments (to finish)
Tuna, Tuna RFMOs, Tuna Atlas, Map, Time series, Catches by Species paul.taconet@ird.fr TO BE DEPLOYED
17 FAO Tuna Atlas VRE CREATE_YOUR_OWN_VERSION_OF_THE_TUNA_ATLAS Development This algorithm allows a user to create its own regional or global tuna altas. It takes as input the latest public domain datasets of the five Tuna Regional Fisheries Management Organizations (IOTC|ICCAT|WCPFC|IATTC|CCSBT) stored within the Sardara database. It proposes a set of parameters to customize the computation of the tuna atlas. In addition to the output dataset it returns a set of information regarding the transformations that are made during the processing. Tuna, Tuna RFMOs, Tuna Atlas https://goo.gl/zP3RM4 paul.taconet@ird.fr https://goo.gl/rxw4YO
18 FAO Tuna Atlas VRE LOAD_A_DATASET_IN_SARDARA_DATABASE Development Load a dataset and associated metadata in Sardara Database. The dataset has to be formatted with the harmonized data structure definition which is: for catches: flag|gear|time_start|time_end|area|schooltype|species|catchtype|catchunit|v_catch for efforts : flag|gear|time_start|time_end|area|school|effortunit|v_effort for catch-at-size : flag|gear|time_start|time_end|area|schooltype|species|sex|catchtype|catchunit|size_step|size_min|v_catch . In case one or several dimension(s) (i.e. column(s)) is/are missing in the dataset to upload it/they will automatically be set to NULL. Additional columns rfmo and ocean might exist in the dataset to upload but are not mandatory. Tuna, Tuna RFMOs, Tuna Atlas https://goo.gl/2j2yY2 paul.taconet@ird.fr NA
19 FAO Tuna Atlas VRE Load a code list in Sardara TO BE DEPLOYED Load a code list in Sardara. Tuna, Tuna RFMOs, Tuna Atlas
20 FAO Tuna Atlas VRE Load a mapping between code lists in Sardara TO BE DEPLOYED Load a mapping between code lists in Sardara. Tuna, Tuna RFMOs, Tuna Atlas
21 FAO Tuna Atlas VRE Raise geo-referenced data (catch or effort) to total catch TO BE DEPLOYED Raise georeferenced catch or effort dataset to total catch dataset Tuna, Tuna RFMOs, Tuna Atlas paul.taconet@ird.fr
22 FAO Tuna Atlas VRE Modify spatial resolution of geo-referenced dataset TO BE DEPLOYED Upgrade or downgrade the spatial resolution of a dataset. Tuna, Tuna RFMOs, Tuna Atlas paul.taconet@ird.fr
23 FAO Tuna Atlas VRE Remove or reallocate data located in land or with no area dimension TO BE DEPLOYED Some data might be mislocated: either located on land areas or without any area information. This algorithm allows to remove or reallocate the mislocated data. If reallocation is chosen, it equally distributs the values of these mislocated areas on areas with same dimensions (month|gear|flag|species|schooltype). Tuna, Tuna RFMOs, Tuna Atlas paul.taconet@ird.fr
24 FAO Tuna Atlas VRE Convert catch unit TO BE DEPLOYED Convert catches expressed in number of fishes to weight using factors of conversion defined by IRD. The methodology to compute these factors is available here: https://goo.gl/F7zGGs Tuna, Tuna RFMOs, Tuna Atlas paul.taconet@ird.fr
25 FAO Tuna Atlas VRE Compute a set of indicators from a dataset TO BE DEPLOYED Ouptut is a report containing plots and maps summarizing the dataset Tuna, Tuna RFMOs, Tuna Atlas paul.taconet@ird.fr
26 FAO Tuna Atlas VRE Transform data structure of IOTC catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of IOTC catch-and-effort datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, IOTC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
27 FAO Tuna Atlas VRE Transform data structure of IOTC nominal catch datasets TO BE DEPLOYED This script converts the raw structure definition of IOTC nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, IOTC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
28 FAO Tuna Atlas VRE Transform data structure of ICCAT catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of ICCAT catch-and-effort datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, ICCAT, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
29 FAO Tuna Atlas VRE Transform data structure of ICCAT nominal catch datasets TO BE DEPLOYED This script converts the raw structure definition of ICCAT nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, ICCAT, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
30 FAO Tuna Atlas VRE Transform data structure of IATTC catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of IATTC catch-and-effort datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, IATTC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
31 FAO Tuna Atlas VRE Transform data structure of IATTC nominal catch datasets TO BE DEPLOYED This script converts the raw structure definition of IATTC nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, IATTC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
32 FAO Tuna Atlas VRE Transform data structure of WCPFC catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of WCPFC catch-and-effort datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, WCPFC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
33 FAO Tuna Atlas VRE Transform data structure of WCPFC nominal catch datasets TO BE DEPLOYED This script converts the raw structure definition of WCPFC nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, WCPFC, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
34 FAO Tuna Atlas VRE Transform data structure of CCSBT catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of CCSBT catch-and-effort datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, CCSBT, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
35 FAO Tuna Atlas VRE Transform data structure of CCSBT nominal catch datasets TO BE DEPLOYED This script converts the raw structure definition of CCSBT nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, CCSBT, Data structure harmonization, Data structure transformation paul.taconet@ird.fr
36 FAO Tuna Atlas VRE Transform data structure of CCAMLR catch-and-effort datasets TO BE DEPLOYED This script converts the raw structure definition of CCAMLR nominal catch datasets to the harmonized structure that is used as input of Sardara and the scripts available in the VRE. Tuna, Tuna RFMOs, Tuna Atlas, CCAMLR, Data structure harmonization, Data structure transformation paul.taconet@ird.fr

tuna_atlas_workflow.png (260 KB) Julien Barde, Nov 24, 2016 05:21 AM