Project Task #688

Project WP #686: WP9 - VRE Commons Development [Months: 1-30]

T9.2 Data Analytics Facilities [Months: 1-29]

Added by Franco Zoppi almost 4 years ago. Updated 3 months ago.

Status:ClosedStart date:Dec 18, 2015
Priority:UrgentDue date:Jan 18, 2018
Assignee:Gianpaolo Coro% Done:

96%

Sprint:WP09
Lead beneficiary:1 - CNR Participants:4 - UOA
Milestones:
Duration: 545

Description

Task leader: CNR; Participants: UOA;
This task will develop a set of facilities to be used by VRE implementations for processing data through a large yet extensible set of approaches including statistical, data mining, quantitative analysis, and machine learning methods.
These facilities will be implemented by extending the facilities offered by the gCube technology. In particular, BlueBRIDGE will extend the analytics facilities by enlarging the set of algorithms and methods offered as a service.
State of the art models, commonly used in computational biology and environmental science and engineering, will be added, e.g. Markov Chain Monte Carlo methods, Support Vector Machines, Generalized Additive Models (GAM), Genetic Algorithms for Rule-Set Production (GARP). Moreover, the mechanisms enabling the integration of new algorithms and methods will be strengthened and further simplified by adding an end user facility enabling authorised users to publish and share algorithms. The environment will be further extended by enabling the exploitation of existing workflows discovered through recognised catalogues, e.g. Taverna / Biodiversity Catalogue / myExperiment. Finally, the environment will be enriched with a workflow creation facility enabling to create new methods and algorithms by combining existing ones.


Subtasks

Task #2520: Create an algorithm to publish HTML pages on an Apache se...ClosedGianpaolo Coro

gCube - Feature #2545: Dataminer enhancementsClosedGianpaolo Coro

Task #3078: Enhance SAIClosedGiancarlo Panichi

Task #3081: Workspace enhancement requestsClosedFrancesco Mangiacrapa

D4Science Infrastructure - Task #1854: Tool for usage statistics on DataMinerClosedAndrea Dell'Amico

gCube - Bug #3925: Issue contacting URI Resolver after several Workspace upl...ClosedRoberto Cirillo

Task #3986: Enable WPS interface for the Ichtyop modelClosedGianpaolo Coro

Task #3987: Build an example of data embedding request to WPSClosedGianpaolo Coro

Task #4121: Create an enhanced version of a Statman Algorithm to publ...ClosedGianpaolo Coro

Task #4138: Integrate CCAMLR charts updaterClosedGianpaolo Coro

Task #4171: Support algorithms integration via SAIClosedGianpaolo Coro

Project Activity #4172: Dataminer interface enhancementsClosedGiancarlo Panichi

Project Activity #4185: publishing the "VPA_ICCAT_BFT_E_Retros" algorithm in the ...ClosedRoberto Cirillo

Task #4211: Intersect two layers in VRE using RClosedLevi Westerveld

Project Activity #4195: publishing the "Ichthyop_model_one_by_one" algorithm in t...ClosedGianpaolo Coro

D4Science Infrastructure - Task #4610: Algorithms installation in D4ScienceClosedGianpaolo Coro

Task #4684: Install SPREAD on DataminerClosedEmmanuel Blondel

gCube - Feature #3983: Implement DataMiner client libraryClosedGiancarlo Panichi

gCube - Feature #4713: DataMiner - Add a direct link to the algorithmsClosedGiancarlo Panichi

Task #4745: Shapefiles importing algorithmClosedGianpaolo Coro

Task #4802: Import OSCAR data in the production environmentClosedGianpaolo Coro

Incident #4819: Oscar files not visible on GeoExplorerClosedFrancesco Mangiacrapa

Project Activity #4896: SAI enhancements ClosedGiancarlo Panichi

Project Activity #4899: Dataminer service enhancementsClosedGianpaolo Coro

gCube - Feature #5258: Add link to help on DataminerClosedGiancarlo Panichi

Support #4957: Help for parallization of Fortran, ICCAT step 3 algorithmRejectedJulien Barde

D4Science Infrastructure - Task #5169: Prepare environment to test Generic Worker on DataminerClosedAndrea Dell'Amico

gCube - Feature #5549: Connect to the new Data Transfer Service in DataMinerIn ProgressLucio Lelii

Task #5459: connect virtual workspace with RStudio workspaceClosedValentina Marioli

gCube - Support #5555: Production dependencies update for DataMiner 4.2.0ClosedGianpaolo Coro

D4Science Infrastructure - Task #5590: DataMiner as Generic WorkerClosedAndrea Dell'Amico

D4Science Infrastructure - Incident #5638: Check data availability through URI ResolverRejectedRoberto Cirillo

D4Science Infrastructure - Incident #5645: Service EP Retrieval issueClosedLucio Lelii

D4Science Infrastructure - Incident #5626: SAI does not save filesClosedRoberto Cirillo

Task #5725: CMSY as-a-ServiceClosedGianpaolo Coro

gCube - Feature #5747: WPS Statistical - Migrate to DataMiner ClosedAndrea Dell'Amico

Task #5912: Interact with OBIS through REST APIClosedGianpaolo Coro

Project Activity #5913: CMSY report templateRejectedEnrico Anello

Task #7101: Defining and creating workspaces on Geoserver dynamicallyClosedFabio Sinibaldi

gCube - Task #7102: Transform DataMiner proxy information into Service EP inf...NewRoberto Cirillo

Task #7120: Manage SAI outputClosedGiancarlo Panichi

D4Science Infrastructure - Task #7126: SAI should make user specify the algorithm categoryClosedGiancarlo Panichi

Task #7129: K-Statistics and DataMinerClosedPaolo Scarponi

D4Science Infrastructure - Task #7167: DataMiner interface should report encoded parameters in t...ClosedGianpaolo Coro

Task #7352: Data Miner Installations AlignerClosedPaolo Scarponi

Task #7549: Identify interesting repositories from OpenDOARClosedPanagiota Koltsida

gCube - Bug #7638: oai-pmh harvester portlet static number of records displayClosedNikolas Laskaris

gCube - Bug #7637: OAI-PMH Harvester portlet - Edit buttonClosedNikolas Laskaris

gCube - Task #7639: OAI-PMH harvester portlet enhancementClosedNikolas Laskaris

gCube - Task #7640: OAI-PMH Harvester portlet - Improve Error MessagesClosedNikolas Laskaris

Project Activity #7641: Harvest and Index the following data sources in iSearch VREClosedPanagiota Koltsida

Task #8034: Install tomcat manager on dataminer machinesRejected_InfraScience Systems Engineer

D4Science Infrastructure - Task #8367: Automatize DataMiner TestingClosedLucio Lelii

D4Science Infrastructure - Incident #8942: Check Mono installationClosed_InfraScience Systems Engineer

Task #8952: Make R scripts inherit token and SDI informationClosedLucio Lelii

Task #9274: CSV to NETCDF Conversion SuiteClosedTAHA IMZILEN

Task #9321: DataMiner should work with local IP addressesClosedAndrea Dell'Amico

Task #9660: Activate a VM in the GARR OpenStack instance and configur...ClosedAndrea Dell'Amico

Task #9661: Activate a dataminer instance without public IP address i...ClosedAndrea Dell'Amico

D4Science Infrastructure - Task #9409: Report the computation ID as a metadata in the DataMiner ...NewLucio Lelii

Task #9521: Integrate the CMSY version for NOAA DLM ToolClosedGianpaolo Coro

D4Science Infrastructure - Support #9536: Critical Workspace-SAI issuesClosedValentina Marioli

D4Science Infrastructure - Task #9569: HL: Problem with locksClosedLucio Lelii

D4Science Infrastructure - Task #9570: Metadata information not correctly reported to SAIRejectedValentina Marioli

D4Science Infrastructure - Task #9571: HL: not add extensions to files during the zip creationClosedValentina Marioli

D4Science Infrastructure - Task #9549: Make R scripts inherit user name and VREClosedGianpaolo Coro

Task #9554: Support Vector Machines as a ServiceClosedPaolo Scarponi

D4Science Infrastructure - Task #9583: Represent ASFIS AquaMaps distribution with NetCDF formatClosedPaolo Scarponi

Task #9707: Change OBIS interaction calls in SPDClosedLucio Lelii

Task #9832: Generate climate change and environmental structured data...ClosedGianpaolo Coro

Task #9936: Switch the cloud computing facility for PAIMClosedGianpaolo Coro

D4Science Infrastructure - Task #10003: Solve the issue of Windows line terminators in shell scriptsClosedGianpaolo Coro

Task #10130: Analysis of Climate Change DataClosedGianpaolo Coro

D4Science Infrastructure - Task #10177: Install the CMSY-FAST algorithm in the StockAssessment VREClosedGianpaolo Coro

D4Science Infrastructure - Task #10223: Provide machines for FAO courseClosed_InfraScience Systems Engineer

Task #10475: Rasterize AquaMaps Native 2050 distributionsClosedGianpaolo Coro

D4Science Infrastructure - Task #10479: Cloud Provisioning RequestsClosed_InfraScience Systems Engineer

gCube - Bug #10517: SAI Set MainClosedGiancarlo Panichi

D4Science Infrastructure - Task #10518: Add the CMSY_LEGACY algorithm to the StockAssessment VREClosedPaolo Scarponi

D4Science Infrastructure - Task #10519: Add the CMSY_VECTORIZED algorithm to the StockAssessment VREClosedPaolo Scarponi

Task #10649: Assess the Stock Assessment processesClosedGianpaolo Coro

Task #10705: Add user name to algorithms descriptions in SAI publicationClosedGianpaolo Coro

gCube - Task #10749: DataMiner algorithms sharingClosedGianpaolo Coro

gCube - Task #10750: The algorithms installer should manage a list of users wi...ClosedGianpaolo Coro

gCube - Task #10778: The DataMiner should filter the algorithms depending on t...ClosedLucio Lelii

gCube - Task #10779: The SAI should send information to the Pool Manager about...ClosedGiancarlo Panichi

gCube - Task #10994: SAI should allow deploying in another VREIn ProgressGiancarlo Panichi

D4Science Infrastructure - Task #10813: Make the new Support Vector Machines models available to ...ClosedPaolo Scarponi

D4Science Infrastructure - Task #11047: Install PhotoScan license server on dlib8x.dom0.research-...Closed_InfraScience Systems Engineer

D4Science Infrastructure - Task #11395: Reinstall dlib8x.dom0.research-infrastructures.eu changin...Closed_InfraScience Systems Engineer

Task #11415: Import Argo observation data in the BlueBRIDGE VREsClosedGianpaolo Coro


Related issues

Related to D4Science Infrastructure - Support #511: Publishing raster data via StatMan Closed Sep 01, 2015
Related to gCube - Bug #426: Investigate WPS authentication methods Closed Jul 24, 2015
Related to D4Science Infrastructure - Task #286: Integrate algorithms developed by Brazilian student Closed Jun 23, 2015
Related to D4Science Infrastructure - Task #208: Prepare and process Indian Ocean Tuna Commission catch st... Closed Jun 03, 2015
Related to gCube - Feature #128: Managing users and scopes via token on WPS synch Closed May 19, 2015
Related to gCube - Feature #114: Make the WPS-sychronous service evaluated by the Taverna ... Closed May 18, 2015
Related to gCube - Feature #1129: Explore connection to StatMan from R - WPS clients invest... Closed Oct 20, 2015
Related to gCube - Feature #1132: StatMan algorithm for geolocalising a table from a GRID-C... Closed Oct 20, 2015
Related to gCube - Feature #995: add a group on the StatMan for Vessel Activities Analysis Closed Oct 20, 2015
Related to gCube - Feature #1133: StatMan algorithm for geolocalising a table from a CSquar... Closed Oct 20, 2015
Related to gCube - Feature #1134: StatMan algorithm for associating FAO Areas to a geospati... Closed Oct 20, 2015
Related to gCube - Feature #1135: StatMan algorithm for associating bathymetry to a geospat... Closed Oct 20, 2015
Related to gCube - Feature #1136: StatMan algorithm to analyse vessels trajectories and to ... Closed Oct 20, 2015
Related to gCube - Feature #1222: Plan to release the new StatMan Closed Oct 22, 2015
Related to gCube - Feature #843: Developing a connector to the SeaDataNet interpolation se... Closed Oct 01, 2015
Related to D4Science Infrastructure - Task #1368: Install and configure "Mono" to run EwE stock assessment ... Closed Nov 16, 2015
Related to gCube - Feature #1451: Design and implementation of an importer of community dev... Closed Nov 20, 2015
Related to D4Science Infrastructure - Task #1855: Promote DataMiner for QGIS Closed Dec 18, 2015
Related to D4Science Infrastructure - Task #1215: Install Dataminer in production environment Closed Oct 22, 2015
Related to D4Science Infrastructure - Task #1838: Smart-generic worker should download file by uri-resolver Closed Dec 17, 2015
Related to BlueBRIDGE - Project Activity #1825: WP5 - T5.1 Blue Assessment - Latex / KnitR / markdown Aut... Rejected Dec 15, 2015
Related to BlueBRIDGE - Project Activity #1903: Integrate IRD VPA Workflow Closed Apr 06, 2016
Related to BlueBRIDGE - Project Activity #1983: Publish NetCDF files in the e-Infrastructure Closed Jan 14, 2016
Related to BlueBRIDGE - Task #1461: Design the interface between the Statistical Manager and EwE Closed Sep 06, 2016 Sep 22, 2016
Related to gCube - Feature #1452: Implement a GUI for StatMan Algorithms Importer Closed Mar 24, 2016
Related to BlueBRIDGE - Task #1777: Ecopath taxonomy dataformats Rejected Dec 10, 2015 Oct 01, 2016
Related to BlueBRIDGE - Task #2229: A WPS process to download infrastructure GIS products as ... Closed Feb 10, 2016
Related to BlueBRIDGE - Project Activity #2230: Provide an algorithm to transform CSV files into NetCDF f... Closed Feb 10, 2016
Related to D4Science Infrastructure - Support #2317: Release the Statistical Algorithms Importer Closed Feb 19, 2016
Related to gCube - Feature #2521: Explore the possibility to port the StatMan interface ont... Closed Mar 09, 2016
Related to gCube - Feature #3169: RStudio-Wrapper-portlet Closed Apr 11, 2016 Apr 22, 2016
Related to BlueBRIDGE - Task #3787: Integrate OpenCPU with the e-Infrastructure Closed Apr 26, 2016 May 23, 2016
Related to BlueBRIDGE - Project Activity #3153: Annotation of charts, discussion about identified tools a... Closed Apr 06, 2016
Related to BlueBRIDGE - Task #7149: Estimate alien species spread - the pufferfish use case Closed Feb 17, 2017 Apr 01, 2017
Related to BlueBRIDGE - Project Activity #6508: BiOnym / Output file / Unmapped rows and process info Closed Apr 07, 2017
Related to BlueBRIDGE - Task #7865: Generate Taxa Authority File for WoRMS Closed Apr 07, 2017
Related to D4Science Infrastructure - Task #7900: Generating Darwin Core Archives via SPD Closed Apr 07, 2017
Related to gCube - Bug #8975: The new DataMiner should recover two old features Closed Jun 19, 2017
Related to gCube - Bug #9084: Status Retrieval not working with the new DataMiner Closed Jun 27, 2017
Related to BlueBRIDGE - Task #5559: Add metadata element to get the source code within descri... Rejected Jun 22, 2017 Jun 29, 2017
Related to BlueBRIDGE - Task #6941: Build a multi-core benchmark for RStudio to test resource... Closed Feb 07, 2017
Related to D4Science Infrastructure - Task #9328: SAI should run SAVE+CREATE when pressing the Publish Button Rejected Jul 20, 2017
Related to D4Science Infrastructure - Task #9329: Publish DataMiner logs through nginx Closed Jul 20, 2017
Related to BlueBRIDGE - Task #5557: automatize the process of publication of algorithms on da... Closed Mar 31, 2017
Related to gCube - Feature #9450: DataMiner PoolManager - Improve the staging stage Closed Jul 31, 2017
Related to BlueBRIDGE - Project Activity #9973: Produce climate change scenarios by processing NASA forec... Closed Oct 17, 2017

History

#1 Updated by Franco Zoppi almost 4 years ago

  • Start date changed from Sep 12, 2015 to Sep 01, 2015

#2 Updated by Franco Zoppi almost 4 years ago

  • Due date changed from Feb 28, 2018 to Jan 31, 2018
  • Subject changed from T9.2 Data Analytics Facilities to T9.2 Data Analytics Facilities [Months: 1-29]

#3 Updated by Franco Zoppi almost 4 years ago

  • Status changed from New to In Progress

#4 Updated by Massimiliano Assante almost 4 years ago

  • Assignee set to Gianpaolo Coro

#5 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Support #511: Publishing raster data via StatMan added

#6 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Support #510: Data Transfer on the THREDDS instance added

#7 Updated by Gianpaolo Coro almost 4 years ago

  • Related to deleted (Support #510: Data Transfer on the THREDDS instance)

#8 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Bug #426: Investigate WPS authentication methods added

#9 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Task #286: Integrate algorithms developed by Brazilian student added

#10 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Task #208: Prepare and process Indian Ocean Tuna Commission catch statistics added

#11 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #128: Managing users and scopes via token on WPS synch added

#12 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #114: Make the WPS-sychronous service evaluated by the Taverna Workflow Management System team added

#13 Updated by Massimiliano Assante almost 4 years ago

  • Start date set to Sep 01, 2015

due to changes in a related task

#14 Updated by Massimiliano Assante almost 4 years ago

  • Due date set to Jan 31, 2018

#15 Updated by Gianpaolo Coro almost 4 years ago

September 2015 Activity Report

Agreement at the Kick-off meeting for the activities in T9.2 involved enhancements to the facilities offered by the Statistical Manager and by the Tabular Data Manager services of D4Science. An overview of the D4Science techniques for Data Analytics and of the users and stakeholders currently using them was shown at the meeting, along with a short-term plan. The shown facilities regarded: support to the FAO Tuna Atlas case, analysis methods to practice stock assessment, ecological modelling, biodiversity analysis, geospatial data processing, time series forecasting and data harmonisation using large code lists or authoritative taxonomic repositories.

The short-term plan presented at the Kick-off meeting involved the following activities:

As for infrastructure facilities:
1. Publication of all the StatMan algorithms via WPS [Oct. ’15->Jun ‘16]
2. Production of NetCDF files out of geospatial data in D4Science [Mar ‘16]
3. Estimating best practices of infra usage, stock assessment, biodiversity analysis [Aug ‘16]
4. Enhancing standard description of datasets: SDMX datasets import and export [Feb ‘16]
5. Enhancing StatMan parallel processing capabilities [Oct ’15->Sep ‘16]

As for Community facilities:
1. Integration of BB community models: EwE, ICCAT, FIN, UoA [Sept ’15->2017]
2. Connecting SeaDataNet DIVA interpolation [Jan ‘16]
3. Enhancing social networking to manage experiment-specific discussions [Feb ‘16]
4. Automatic reports production from users’ interactions [Jun ‘16]
5. Supporting offline/online e-learning and courses [Sep ’15->2017]

In the month of September 2015, activity on the Statistical Manager involved the following sub-activities:

  • Developing and releasing an algorithm to publish raster data in the infrastructure. This activity also involved enhancements on the Data Transfer services.

  • Developing an authentication method for the WPS interface to the StatMan algorithms that was compliant with external clients and software (e.g. QGIS, Chrome etc.).

  • Integrating two algorithms developed by Universidade Federal de Mato Grosso (Brazil) which normalise precipitation data

  • Refining a Time Series Forecasting process and applying it to Indian Ocean Tuna Commission data

Activity in September reports also that, even if the Taverna software is supporting WPS, the Taverna team is not going support the CNR team with assessment and testing activity. Nevertheless, this assessment has been substituted by a WPS service testing using Quantum GIS.

#16 Updated by Pasquale Pagano almost 4 years ago

  • % Done changed from 0 to 10

#17 Updated by Gianpaolo Coro almost 4 years ago

  • Start date set to Sep 01, 2015

due to changes in a related task

#18 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1129: Explore connection to StatMan from R - WPS clients investigated added

#19 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1132: StatMan algorithm for geolocalising a table from a GRID-CWP column added

#20 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #995: add a group on the StatMan for Vessel Activities Analysis added

#21 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1133: StatMan algorithm for geolocalising a table from a CSquare column added

#22 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1134: StatMan algorithm for associating FAO Areas to a geospatially explicit table added

#23 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1135: StatMan algorithm for associating bathymetry to a geospatially explicit table added

#24 Updated by Gianpaolo Coro almost 4 years ago

  • Related to Feature #1136: StatMan algorithm to analyse vessels trajectories and to estimate CPUE, fishing hours and fishing activity added

#25 Updated by Gianpaolo Coro over 3 years ago

  • Related to Feature #1222: Plan to release the new StatMan added

#26 Updated by Gianpaolo Coro over 3 years ago

  • Related to Feature #843: Developing a connector to the SeaDataNet interpolation service added

#27 Updated by Gianpaolo Coro over 3 years ago

October 2015 Activity Report

Activity in T9.2 regarded StatMan.

On client side:

  • a WPS connector for R was developed to use BlueBRIDGE processes from this programming language

On service side:

  • A process to produce uniform environmental maps from satellite or in situ data was developed, which uses the SeaDataNet DIVA interpolation web-service. This algorithm was successfully tested with Copernicus data
  • Porting of Vessels Transmitted Information processes onto StatMan was started
  • The implementation of an algorithm to produce GRID-CWP codes from longitude and latitude columns in a table was completed and discussed with FAO
  • A group of algorithms collecting processes for vessels data was added to StatMan in the production environment
  • A new version of StatMan solving well known issues has been released and installed
  • Another version of StatMan better supporting VREs scopes management is under testing

#28 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1368: Install and configure "Mono" to run EwE stock assessment algorithms added

#29 Updated by Gianpaolo Coro over 3 years ago

#30 Updated by Gianpaolo Coro over 3 years ago

  • Related to Feature #1451: Design and implementation of an importer of community developed algorithms onto StatMan (StatMan Algorithms Importer) added

#31 Updated by Gianpaolo Coro over 3 years ago

November 2015 Activity Report

  1. The SeaDataNet DIVA interpolation system has been finalised and the release of this component in production environment has started #843
  2. Porting of the old VTI algorithms onto StatMan has been completed #1132 #995 #1133 #1134 #1135 #1136
  3. A new version of StatMan has been released, which solves well-known bugs of the previous version and makes the system much more stable
  4. A novel version of StatMan is under testing and release #1222
  5. Support for integrating Ecopath with Ecosym has been given: the operative system environment to integrate EwE with StatMan has been prepared #1368
  6. Work is still being managed to understand how to develop the interface between EwE and StatMan #1441
  7. The design and implementation of a new system to allow community scientists to upload algorithms by themselves has started #1451

#32 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1855: Promote DataMiner for QGIS added

#33 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1854: Tool for usage statistics on DataMiner added

#34 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1215: Install Dataminer in production environment added

#35 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1838: Smart-generic worker should download file by uri-resolver added

#36 Updated by Gianpaolo Coro over 3 years ago

  • Related to Project Activity #1825: WP5 - T5.1 Blue Assessment - Latex / KnitR / markdown Automated Reports for Bluefin Tuna Stock Assessment Workflow added

#37 Updated by Gianpaolo Coro over 3 years ago

#38 Updated by Gianpaolo Coro over 3 years ago

December 2015 Activity Report

  1. The SeaDataNet Interpolation algorithm has been released and a video has been produced, which demonstrates how to process Copernicus data and publish/share a map on the e-Infrastructure #843
  2. A new version of the Statistical Manager has been completed and tested, which enhances performance and solves several bugs #1223
  3. Activity on EwE integration has continued. IRD, CNR and ENG agreed on an integration strategy #1441 #1368
  4. The design and implementation of a new system allowing community scientists to upload algorithms by themselves has continued #1451
  5. DataMiner has been installed in the production environment as a scalable data processing system. FishBase requests will be managed by this system in the future #1215. DataMiner is also going to be promoted as a native tool in the QuantumGIS software #1855

#39 Updated by Gianpaolo Coro over 3 years ago

#40 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1461: Design the interface between the Statistical Manager and EwE added

#41 Updated by Gianpaolo Coro over 3 years ago

  • Related to Feature #1452: Implement a GUI for StatMan Algorithms Importer added

#42 Updated by Gianpaolo Coro over 3 years ago

#43 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #1777: Ecopath taxonomy dataformats added

#44 Updated by Gianpaolo Coro over 3 years ago

January 2016 Activity Report

  • CNR continued enhancing the interface of an application (SAI) allowing community scientists to upload algorithms by themselves #1451
  • CNR supported the integration of the Ecopath with Ecosim software with the Statistical Manager #1461
  • One algorithm to publish raster data was provided via WPS and experimented with IRD #1983
  • One algorithm to retrieve information from FishBase tables was provided via WPS and experimented with IRD #1777
  • A tool for producing statistics about the service usage is being investigated #1854
  • Integration and experimentation of the VPA Workflow by IRD, Ifremer and ICCAT was started #1903

#45 Updated by Gianpaolo Coro over 3 years ago

  • Related to Task #2229: A WPS process to download infrastructure GIS products as ESRI GRID files added

#46 Updated by Gianpaolo Coro over 3 years ago

#47 Updated by Gianpaolo Coro over 3 years ago

  • Related to Support #2317: Release the Statistical Algorithms Importer added

#48 Updated by Gianpaolo Coro over 3 years ago

February 2016 Activity Report

  • CNR worked on finalising the Statistical Algorithms Importer #1451.
  • CNR started integrating the VPA workflow with the help of IRD. A first version was made available in the prototyping environment #1903.
  • A process to download GIS products as Raster ESRI-GRID files has been implemented and released #2229.
  • CNR is investigating with FAO and IRD, possible strategies to transform geospatially explicit datasets into NetCDF files #2230.
  • The Knitr compiler was released in the production environment #1825.
  • A new and more efficient version of StatMan (2.0) was released in the production environment #1222.

#49 Updated by Gianpaolo Coro over 3 years ago

  • Related to Feature #2521: Explore the possibility to port the StatMan interface onto Dataminer added

#50 Updated by Massimiliano Assante over 3 years ago

#51 Updated by Gianpaolo Coro over 3 years ago

March 2016 Activity Report

  • CNR finished releasing a process to download and extract GIS information from the infrastructure as ESRI GRID files (ASCII format) #2229
  • CNR deployed new versions of the Statistical Manager and Dataminer in the production environment, endowed with more algorithms #1215
  • A tool to analyse users' computations details was investigated and tested #1854
  • The VPA Workflow was integrated using SAI to demonstrate the feasibility to run this process in the infrastructure #1903
  • The Statistical Algorithms Importer was released in the production environment #2317
  • The feasibility of a large activity to move the Statistical Manager completely to WPS was investigated #2521
  • Dataminer was endowed with functions to manage provenance using the infrastructure Workspace, with provenance information managed using the Prov-O ontology #2545
  • Workspace capabilities have been extended to support provenance management #2546

#52 Updated by Giancarlo Panichi over 3 years ago

  • Due date set to Apr 21, 2016

due to changes in a related task

#53 Updated by Giancarlo Panichi over 3 years ago

  • Due date set to Mar 30, 2016

due to changes in a related task

#54 Updated by Gianpaolo Coro about 3 years ago

  • Related to Task #3787: Integrate OpenCPU with the e-Infrastructure added

#55 Updated by Gianpaolo Coro about 3 years ago

  • Related to Project Activity #3153: Annotation of charts, discussion about identified tools and possible options added

#56 Updated by Gianpaolo Coro about 3 years ago

April 2016 Activity Report

  • CNR investigated the possibility to integrate the e-Infrastructure computational facilities with the OpenCPU technology. This will allow Web developers to simply integrate javascript libraries to interact with Dataminer #3787
  • CNR developed a wrapper portlet for RStudio that allows users to work directly with this IDE without passing through TabMan (as the previous approach required) #3169
  • CNR continued developing the porting of the StatMan interface onto DataMiner #2521
  • An algorithm to publish HTML pages on a public website hosted by the infrastructure, was released on the BlueBRIDGE portal #2520
  • CNR supported the BlueBRIDGE users' and developers' community with tools and discussions to manage charts annotations #3153 and publication of NetCDF files #1983

#57 Updated by Massimiliano Assante about 3 years ago

  • Related to deleted (Task #1854: Tool for usage statistics on DataMiner)

#58 Updated by Massimiliano Assante about 3 years ago

  • Related to deleted (Project Activity #1825: WP5 - T5.1 Blue Assessment - Latex / KnitR / markdown Automated Reports for Bluefin Tuna Stock Assessment Workflow)

#59 Updated by Massimiliano Assante about 3 years ago

Dear GP, as agreed during the last #PEC https://support.d4science.org/issues/2549 the activities we report in wp9 tasks as related tickets should be migrated to sub-tasks.

Please migrate the current related activities that are not closed as sub-tasks (where possible). Please keep this change in mind for the future and upcoming activities reported under wp9 tasks T9.2

#60 Updated by Gianpaolo Coro about 3 years ago

May 2016 Activity Report

  • CNR worked to make the RStudio environment directly accessible from the portal, using the same authorization methods of the portal and generally of the infrastructure #3169 #2516
  • The OpenCPU service was integrated with Dataminer, in order to provide a javascript connection to the infrastructure algorithms. This was achieved by porting the Dataminer-WPS-R-connector on OpenCPU. FAO is currently integrating an algorithm to demonstrate charts production on CCAMLR (Antarctic) data #3787 using this technology. This activity came after a sequence of discussions with FAO and IRD about the best technology to use #3153
  • Large effort was spent on the refinement of Dataminer, to let it communicate with the Workspace #2545, in order to manage provenance, and on the porting of the StatMan Web interface to Dataminer #2521
  • The Ichtyop model was ported on StatMan and was also endowed with a WPS interface, by porting it on Dataminer #3986
  • Support has been provided to IRD and FAO to integrate their processes via SAI #4171

#61 Updated by Gianpaolo Coro about 3 years ago

June 2016 Activity Report

  • CNR helped in the development of a connection between a FAO Web application and Dataminer for a CCAMLR demo tool #3787
  • The porting of the StatMan interface on Dataminer finished #2521
  • RStudio was endowed with https communication #3169
  • The discussion about charts annotation was concluded by selecting OpenCPU as the bridge for Javascript applications #3153
  • CNR gave support in a number of WP5/WP6/WP7 related issues about algorithms development (e.g. #4211, #3563, #3158, #1777, #3835, #4211, #2289, #2344)

#62 Updated by Gianpaolo Coro about 3 years ago

  • Due date changed from Jun 30, 2016 to Jul 07, 2016

due to changes in a related task

#63 Updated by Gianpaolo Coro almost 3 years ago

  • Due date set to Jul 28, 2016

due to changes in a related task

#64 Updated by Gianpaolo Coro almost 3 years ago

July and August 2016 Activity Report

  • The OSCAR NASA data were ported to the production environment to be used in drifting experiments #4802
  • A process to import Shapefiles within the infrastructure was finished and proposed to Grid-A for experiments #4745
  • An operation to rapidly link algorithms and get their user interface was developed #4713
  • The implementation of a client library for Dataminer resembling the Statman one was started #3983
  • Activity to port SPREAD to Dataminer was started #4684
  • ~30 algorithms provided by the BB community were integrated with Dataminer using SAI #4610
  • A parallelised version of the IRD Ichthyop model was implemented and integrated with Dataminer #4195
  • Active support was given to Grid-A and FAO in the development of a script to intersect WFS layers #4211
  • The ICCAT Blue Fin Tuna processes were ported to the production environment #4185
  • The Dataminer interface now manages polygons drawing #4172
  • The StatMan interface was completely ported on Dataminer #2521

#65 Updated by Massimiliano Assante almost 3 years ago

  • Due date set to Aug 30, 2016

due to changes in a related task

#66 Updated by Gianpaolo Coro almost 3 years ago

September 2016 Activity Report

  • SPREAD has been definitely ported on Dataminer #4684
  • The ICHTHYOP model has been ported to the production environment #4195
  • New important features have been added to the Dataminer portlet: bounding boxes selection using interactive maps, time and date selection panels #4172
  • Enhancements to Dataminer have been added, i.e. producing logs as output of any R process, porting of the Generic Worker on Dataminer #4899
  • Support to the installation and enhancement of algorithms has been provided #4610 #4957
  • The Ecopath with Ecosym algorithm has been endowed with feature to alert a user via email #4924

#67 Updated by Gianpaolo Coro over 2 years ago

October 2016 Activity Report

  • The GenericWorker for Cloud computations has been ported to DataMiner #5590 #5169
  • The installation of DataMiner has been fully automatized #5555
  • Activity to connect RStudio with the Workspace has started #5459
  • Activity to use a new DataTransfer service in computations has started #5549
  • An online helper has been added to DataMiner #5258
  • Logs are now produced out of all the SAI algorithms #4899
  • A DataMiner client library is under development #3983
  • About 30 new algorithms have been installed in the production environment #4610

#68 Updated by Gianpaolo Coro over 2 years ago

November 2016 Activity Report

  • CMSY (Oceana version) has been integrated as-a-Service #5725
  • A report template has been defined based on the CMSY process #5913
  • Absence location estimation has been passed to the OBIS REST API #5912
  • The FishBase requests have been all moved to DataMiner #5747
  • SAI has been bug fixed #5626
  • Data availability for processes has been investigated #5638
  • Dataminer now uses several Generic Worker, also from the EGI infrastructure #5590 #5169
  • The Workspace has been integrated with the RStudio environment #5459

#69 Updated by Gianpaolo Coro over 2 years ago

December 2016 and January 2017 Activity Report

  • The interaction between the Workspace and RStudio has been released #5459
  • Issues related to infrastructure resources retrieval by DataMiner have been solved #5645
  • The DataMiner client library has been released, to be used by other services like TabMan #3983

#70 Updated by Gianpaolo Coro over 2 years ago

  • Related to Task #7149: Estimate alien species spread - the pufferfish use case added

#71 Updated by Gianpaolo Coro over 2 years ago

February 2017 Activity Report

  • A FAO use case on the distribution of the pufferfish in Mediterranean sea has been supported. The work is still in progress #7149
  • A general process to estimate agreement between a number of classifiers has been implemented, based on Kappa Statistics #7129
  • Enhancements and refinements of the DataMiner interface have been made #7167
  • SAI has been enhanced by allowing users to specify the algorithm category #7126 and by better managing outputs definitions #7120
  • The GIS-enabling libraries have been extended to support the creation of classifications (workspaces) for imported shapefiles #7101.

#72 Updated by Gianpaolo Coro over 2 years ago

#73 Updated by Gianpaolo Coro over 2 years ago

  • Related to Task #7865: Generate Taxa Authority File for WoRMS added

#74 Updated by Gianpaolo Coro over 2 years ago

March 2017 Activity Report

  • In order to support WP7 activities, CNR enhanced and released a shapefiles publication process with option to indicate a GIS workspace for the generated map #7101
  • Activity to support the FAO T7.3 use case for the pufferfish spread in the Mediterranean Sea, continued with important results #7149
  • An R process to invoke BiOnym was put in place, which will allow FAO to evaluate the match between ASFIS and WoRMS #6508
  • A Big Data generation process has been started to produce Taxonomic Authority Files from WoRMS #7865

#75 Updated by Gianpaolo Coro over 2 years ago

  • Related to Task #7900: Generating Darwin Core Archives via SPD added

#76 Updated by Gianpaolo Coro about 2 years ago

April 2017 Activity Report

  • Activity to support the FAO T7.3 use case for the puffer fish spread in the Mediterranean Sea continued with important results, presented at FAO in a seminar #7149
  • An R process to invoke BiOnym was enhanced, which allows FAO to evaluate the match between ASFIS and WoRMS #6508. This activity also required effort to generate a large repository of taxonomic names representations of the WoRMS entries #7900, which also required interaction with the WoRMS team #7865
  • The OAI-PMH Harvester portlet was analysed for enhancements #7640, in particular to harvest and index a number of data sources for the iSearch VRE #7641 #7549
  • A new aligner process has been put in place to synchronise several DataMiners. This process is currently used to rapidly install an algorithm in the production VREs #7352

#77 Updated by Gianpaolo Coro about 2 years ago

May 2017 Activity Report

  • Enhancements on DataMiner were performed in order to automatise the platform tests #8367 and the deployment process of the service #8034
  • Interaction with FAO to make taxonomic searches in the WoRMS repository through BiOnym has gone forward, involving researchers from Univ. of Washington #7865
  • The process used to estimate marine invasive species is under evaluation by an expert, who periodically interacts with the Task working group #7149
  • A process to generate NeTCDF-CF files out of CSV files is under development and will be presented at the next TCOM in June #2230

#78 Updated by Gianpaolo Coro about 2 years ago

  • Related to Bug #8975: The new DataMiner should recover two old features added

#79 Updated by Gianpaolo Coro about 2 years ago

  • Related to Bug #9084: Status Retrieval not working with the new DataMiner added

#80 Updated by Gianpaolo Coro about 2 years ago

  • Related to Task #5559: Add metadata element to get the source code within describeProcess query added

#81 Updated by Gianpaolo Coro about 2 years ago

  • Related to Task #6941: Build a multi-core benchmark for RStudio to test resource saving added

#82 Updated by Gianpaolo Coro about 2 years ago

June 2017 Activity Report

  • A DataMiner algorithm to transform CSV files into NetCDF files has been implemented in a first version #2230
  • DataMiner has been enhanced to connect to the new Data Transfer service #5549
  • Re-engineering of DataMiner has been started, in order to meet also automatic deployment requirements #8975 #9084 #5559
  • An R Studio benchmark process has been developed to implement restrictions to resources usage by users #6941
  • It is being discussed how to make R scripts inherit information from the e-Infrastructure #8952
  • Activity to adjust the Mono installation has been done in order to support Windows-compiled algorithms #8942

#83 Updated by Gianpaolo Coro about 2 years ago

  • Related to Task #9328: SAI should run SAVE+CREATE when pressing the Publish Button added

#84 Updated by Gianpaolo Coro about 2 years ago

  • Related to Task #9329: Publish DataMiner logs through nginx added

#85 Updated by Gianpaolo Coro almost 2 years ago

  • Related to Task #5557: automatize the process of publication of algorithms on dataminer added

#86 Updated by Gianpaolo Coro almost 2 years ago

  • Related to Feature #9450: DataMiner PoolManager - Improve the staging stage added

#87 Updated by Gianpaolo Coro almost 2 years ago

July-August 2017 Activity Report

Most of the activities have concentrated on the engineering of several DataMiner facilities:

  • Publishing the live logs through nginx #9329
  • Automatisation of testing #8367
  • Enhancing the engineering of the Service #8975 #8952
  • Making SAI more robust to users' interactions #9328
  • Automatisation of the SAI-integrated processes #5557 #9450

  • Four processes to transform CSV files into NetCDF files have been released, which will be suggested to the AquaMaps Consortium as a means to publish their data #2230 #9274

  • A new CMSY version developed by NOAA and FAO has been integrated with DataMiner #9521

  • Support Vector Machines are now part of the general-purpose machine learning tools provided by DataMiner #9554

Ongoing activities:

  • Report of the computation ID in the DataMiner output#9409
  • Publishing the source code of an integrated algorithm #5559
  • Make R Scripts inherit user name and VRE information #9549
  • Make DataMiner work with non-public IP addresses #9321

#88 Updated by Valentina Marioli almost 2 years ago

  • Due date changed from Mar 24, 2017 to Sep 01, 2017

due to changes in a related task

#89 Updated by Andrea Dell'Amico almost 2 years ago

  • Due date changed from Mar 24, 2017 to Sep 11, 2017

due to changes in a related task

#90 Updated by Gianpaolo Coro almost 2 years ago

September 2017 Activity Report

  • Most of the effort has been spent at making the new version of DataMiner available in the production VREs, including all the enhancements developed in the last months;
  • A first version of the automatic algorithm deployment system has been released and installed in the production environment #5557;
  • Inheritance of infrastructure information for R Scripts has been enhanced: DataMiner now passes information about the user name, the VRE, and the user token #9549 #8952
  • Maps of climate change have been produced by processing environmental data from several sources #9832
  • Support has been given to WP5 to make CMSY available under two versions to a FAO Stock Assessment interface #9521
  • Technological support to the invasive puffer fish prediction use case has been given. The activity has been validated both by FAO and two external scientists. A scientific paper has been written and sent to the Ecological Modelling Journal #7149;
  • Connection to OBIS has been enhanced in the Absence Records Estimation algorithm, through the passage to the OBIS API v2 #9699
  • CNR has been also spending effort to make 11,600 AquaMaps Native distributions, hosted by the e-Infrastructure, available as NetCDF files and to propose a transformation service to the AquaMaps Consortium #9583

#91 Updated by Gianpaolo Coro almost 2 years ago

#92 Updated by Gianpaolo Coro over 1 year ago

October 2017 Activity Report

  • Long-term climatic forecasts have been analysed, as published by AquaMaps #9832 and NASA #9973, with a final production of 11 environmental parameters time series that give insight about global climate change under different scenarios of greenhouse gases emission and resources exploitation.
  • A time series analysis on different ocean areas at global scale has revealed interesting shared properties and similarities between these locations #10130. The resulting data can be of support to general Blue Growth analyses. This will be discussed at the "Expert Meeting on Climate Change and Fisheries in the Mediterranean and Black Sea" on December.
  • Enhancements and support have been provided for SAI, e.g. in the management of black boxes when using the Windows OS #10003
  • The transformation and publication of 11,600 AquaMaps Native distributions into NetCDF files has been completed #9583
  • A new version of the CMSY Stock Assessment algorithm has been installed, which is 10 times faster than the legacy algorithm, although with coarser results #10177

The PAIM VRE has been endowed with a powerful cloud computing cluster that strongly enhanced the performance of the integrated algorithms #9936
The possibility to use private IP machines for cloud computing has been verified and a solution has been implemented #9661 #9660 #9321*

#93 Updated by Gianpaolo Coro over 1 year ago

November 2017 Activity Report

  • A large provisioning and testing activity has been conducted to enrich the computational cluster to support computations and courses #10479 #10223
  • A transformation process to rasterize and publish the AquaMaps Native 2050 distributions through Thredds is ongoing #10475
  • A fast version of CMSY developed by NOAA has been integrated with the DataMiner and provided to the FAO Stock Assessment course #10177
  • The Climate Change data forecasts hosted by BlueBRIDGE have been enriched with NASA precipitations data between 1950 and 2100 #9832 #9973
  • Enhancements to the SAI interface have been made to meet users' requirements #10187

#94 Updated by Gianpaolo Coro over 1 year ago

  • Due date changed from Feb 15, 2018 to Dec 18, 2017

due to changes in a related task

#95 Updated by Gianpaolo Coro over 1 year ago

  • Due date changed from Nov 15, 2017 to Jan 18, 2018

due to changes in a related task

#96 Updated by Gianpaolo Coro over 1 year ago

December 2017 Activity Report

  • Enhancements to the SAI interface have been added to enhance user experience #10187. In particular, adjustments like (i) hiding the software creation process in the publication button, (ii) reordering the interface tabs, (iii) managing Workspace locking issues #10199 etc. were made after collecting feedback from users;
  • The Support Vector Machines training and projection processes have been fixed and enhanced to meet requirements of ecological modelling problems #10813
  • A plan to add private sharing of algorithms between users in a VRE has been developed and started, which impacts several services of the infrastructure but has large beneficial impact on the infrastructure users #10749
  • Indexing of the DataMiner algorithms on the GeoNetwork is being investigated #10704
  • A large Cloud computing system has been setup for the DataMiner in the high-performance production VREs
  • Assessment of the Stock Assessment processes integrated in BlueBRIDGE for the WECAFC case has been conducted #10649

#97 Updated by Gianpaolo Coro over 1 year ago

January - February 2018

  • Enhancements to the SAI interface have been added as a continuation of the activity of December #10994 #10187, e.g. the publisher's user name is now reported in the algorithm description #10779, public/private algorithms sharing is enabled #10778 #10750 #10705, work-in-progress regards deploying an algorithm in "another" VRE from SAI #10994
  • The way to index DataMiner algorithms on GeoNetwork is still under study with some options explored during these months #10704
  • An algorithm to automatically build virtual meshes and Web applications from a set of photos has been developed and used in a course #11047 #11205, as a way to liaise with other projects and set the bases of possible new projects.

#98 Updated by Andrea Dell'Amico over 1 year ago

  • Due date changed from Jan 18, 2018 to Mar 07, 2018

due to changes in a related task

#99 Updated by Pasquale Pagano over 1 year ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF