Project WP #697

WP10 - Interfacing Infrastructures [Months: 1-30]

Added by Franco Zoppi about 4 years ago. Updated about 1 year ago.

Status:ClosedStart date:Mar 07, 2016
Priority:UrgentDue date:Jun 01, 2017
Assignee:Gabriele Giammatteo% Done:

100%

Sprint:WP10
Lead beneficiary:3 - ENG Budget Effort (PM):77.00
Milestones:
Duration: 324

Description

Objectives
This work package extends and consolidates the gCube technology by delivering a rich array of facilities enabling the integration of existing infrastructures, including computing and data infrastructures, into a coherent and organised federation that can be easily managed by guaranteeing the operation autonomy of the single resources. Outcomes of this work package serves as basis for the development of functional facilities produced in WP9 as well as vertical solutions developed throughout WP5-8.

Description of work and role of partners
ENG, CNR, UOA


Subtasks

Project Task #698: T10.1 Integrate Computing Infrastructure Resources [Month...ClosedNunzio Andrea Galante

Task #3919: Federted Hosting Node Manager (FHNManager)ClosedNunzio Andrea Galante

Task #3920: FHNManager serviceClosedNunzio Andrea Galante

Task #3921: FHNManager PortletClosedFabio Sinibaldi

Task #4974: FHNManager - Occopus IntegrationClosedNunzio Andrea Galante

Project Task #699: T10.2 Integrate Data Infrastructure Resources [Months: 1-29]ClosedLucio Lelii

Task #2349: Interoperability with the European Data Portal Rejected

Task #4975: Implement an importer for OSCAR datasetRejectedPaolo Fabriani

Task #8713: update OSCAR NetCDFClosedPaolo Fabriani

Project Task #700: T10.3 Federated Resources Management [Months: 1-29]ClosedLuca Frosini

Task #3473: Assess OrientDB for information systemClosedLuca Frosini

Task #3474: Install 3 instances of OrientDB to test dserverClosed_InfraScience Systems Engineer

Task #3476: Test OriendDB Embedded FunctionClosedLuca Frosini

Task #3477: Test dserver performaces and qualityClosedLuca Frosini

Task #3478: Test OriendDB modelClosedLuca Frosini

Task #3932: Test OrientDB Hooks And TriggerClosedLuca Frosini

Task #3933: Test OrientDB Cluster SelectionClosedLuca Frosini

Task #3934: Test OriendDB Live QueryClosedLuca Frosini

Task #3935: Test OriendDB Security to use a Single DB for Multiple Co...ClosedLuca Frosini

Task #3936: Define IS Resource on OrientDB ModelClosedLuca Frosini

Task #3956: Improve Accounting for enhanced integration with ISClosedLuca Frosini

Task #4026: Information SystemClosedLuca Frosini

Task #4023: Define IS ResourcesClosedLuca Frosini

Project Activity #6605: Enable third-party applications to obtain limited access ...ClosedMassimiliano Assante

Project Activity #4940: UserProfile ServiceClosedMassimiliano Assante

gCube - Feature #5143: smartgears service authentication portletClosedMassimiliano Assante

Task #6707: New oauth 2.0 web service to perform the "code to token"...ClosedCostantino Perciante

Task #6708: Provide information to execute code before smartgears' to...ClosedLucio Lelii


Related issues

Related to BlueBRIDGE - Key Performance Indicator #967: Blue Commons: Integrated e-Infrastructures Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Key Performance Indicator #968: Blue Commons: Integrated Repositories Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Key Performance Indicator #969: Blue Commons: Integrated Data Publishers Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Key Performance Indicator #970: Blue Commons: Integrated Datasets Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Key Performance Indicator #971: Blue Commons: Accessible Records Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Key Performance Indicator #972: Blue Commons: Accessible Maps Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #978: 2. Depending technologies / specifications get abandoned Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #980: 4. Delivered systems miss functional or non-functional (p... Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #982: 6. A partner / body underperforms, defaults or faces othe... Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #983: 7. Domain or project’s concepts become obsolete or irrele... Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #984: 8. Delays in production of deliverables, be it interim on... Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Risk #986: 10. Lack of required competences for the completion of pr... Closed Oct 12, 2015 Feb 28, 2018
Related to BlueBRIDGE - Task #1340: Move CITE's OpenSource code into a separate publicly acce... Closed Nov 11, 2015 Jan 20, 2016

History

#1 Updated by Franco Zoppi about 4 years ago

  • Due date changed from Feb 28, 2018 to Jan 31, 2018

due to changes in a related task

#2 Updated by Franco Zoppi about 4 years ago

  • Start date changed from Sep 12, 2015 to Sep 01, 2015

due to changes in a related task

#3 Updated by Franco Zoppi about 4 years ago

  • Status changed from New to In Progress

#4 Updated by Massimiliano Assante almost 4 years ago

  • Assignee set to Gabriele Giammatteo

#5 Updated by Gabriele Giammatteo almost 4 years ago

September 2015 Activities

In T10.1:
* at the KO meeting the various aspects to be considered in the federation of infrastructures have been presented (e.g. security, monitoring, accounting, ...). For some of the aspects also an initial analysis of different alternative solutions have been draft. This topic needs to be further discussed with the task participants in the next f2f meeting.
* It is under investigation if/how to extend/generalize the work done so far in the context of EGI-ENGAGE for the lifecycle management of VMs on external providers.

In T10.2:
* At the KO meeting the state of the art of gCube technology with respect to seamless access to heterogeneous data across data sources has been analysed.
* the resource model characterising gCube resources has been analysed and limitations have been identified.

In T10.3:
* Dedicated presentations and discussions occurred during the KO meeting. An initial understanding of the Task goal and expectations has been developed. An initial understanding of WP5-WP8 needs has been acquired.
* New Accounting system has been integrated in Storage Manager and in Smartgears and gCore (the service invocation are now accounted transparently for the developers).
* New Authorization token mechanism has been developed and integrated in Smartgears and gCore (the token is transparently validated and allows to identify user/service which has originated the service call. The validation use a blacklist to block the unauthorized service call).
* New Resource Model activities has been started.

Next six months planned activities
* produce a set of scenario stressing the charasteristics that the integration/federation of infrastructures should support. They will be presented and discussed at the next f2f meeting (Athens)
* to liaise with WP5-WP8 to collect a list of Data Resources to be integrated / managed;
* to further analyse & describe the major limitations of the current Resource Model and start discussing the new Resource Model;
* to further identify and document the major limitations of the current accounting solution;

More activities will be identified during next f2f meeting in Athens (Nov '15)

#6 Updated by Pasquale Pagano almost 4 years ago

#7 Updated by Pasquale Pagano almost 4 years ago

#8 Updated by Pasquale Pagano almost 4 years ago

#9 Updated by Pasquale Pagano almost 4 years ago

#10 Updated by Pasquale Pagano almost 4 years ago

#11 Updated by Pasquale Pagano almost 4 years ago

#12 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #978: 2. Depending technologies / specifications get abandoned added

#13 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #980: 4. Delivered systems miss functional or non-functional (performance, scalability, etc.) requirements added

#14 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #982: 6. A partner / body underperforms, defaults or faces other severe operation issues added

#15 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #983: 7. Domain or project’s concepts become obsolete or irrelevant during project’s lifetime added

#16 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #984: 8. Delays in production of deliverables, be it interim ones (with dependencies) or final ones added

#17 Updated by Pasquale Pagano almost 4 years ago

  • Related to Risk #986: 10. Lack of required competences for the completion of project’s tasks added

#18 Updated by Gabriele Giammatteo almost 4 years ago

October 2015 Activities
* analysis and design of integration with cloud providers
* occi-lib made available and documented
* some data catalogues analysed: B2Find, DataONE, European Data Portal with the objective of understanding how data catalogues works and are amanaged by the other projects
* New IS Resource model, functional and non-functional requirements collected
* Study of best practices to deploy CouchDB in High Availability to support accounting and authorization non functional requirements

#19 Updated by George Kakaletris almost 4 years ago

  • Related to Task #1340: Move CITE's OpenSource code into a separate publicly accessible repository added

#20 Updated by Gabriele Giammatteo almost 4 years ago

November 2015 Activities

In Task T10.1
- occi-lib has been extended to use JClouds library to cover more cloud providers

In Task T10.2
- continued the study on data catalogues listed in the previous ticket, going deep on the European Data Portal repository.
- a first draft of facets for the representation of resources in the new IS has been completed with contributions from all partners.

In Task T10.3
- in the context of new accounting subsystem, the solution has been made more modular and generalized. New persistence implementations for CouchDb and CouchBase have been implemented.
- the new Authorization Framework has been released.
- started an activity of comparison between CouchDb, CouchBase and MongoDB. This activity is also useful for new IS.

#21 Updated by Gabriele Giammatteo over 3 years ago

December 2015 Activities

In Task T10.1, occi-lib has been tested with Amazon (using new JClouds connector)

In Task T10.2, a list of specifications for facets and resources have been produced with the collaboration of all technical partners.

In Task T10.3:

  • the new accounting framework received some updates in order to support more persistence technologies (i.e. CouchBase and MongoDB)
  • the new authorization framework has been released and integrated in both gCube containers gCore and Smartgears
  • the undergoing analysis of NOSql databases continued (in particular CouchDb <-> CouchBase <-> MongoDB)
  • StatefulResource and Factory patterns support is under implementation on Smartgears

#22 Updated by Gabriele Giammatteo over 3 years ago

January 2016 Activities

  • In the context of integration with other infrastructures (e.g. EGI's FedCloud), the design and implementation of a new gCube servcie (FHNManager) and a portlet to manage gHNs hosted on external infrastructures has started.
  • Authorization framework improved and integrated in SmartGears and gCore.
  • Stateful resource and factory pattern support has been developed and will be integrated in Smartgears in February 2016.
  • Technical presentations for the 1st TCOM have been prepared related to resource model, authz and accounting frameworks, stateful resources.
  • Milestone MS8 has been achieved and documented (#703)

#23 Updated by Gabriele Giammatteo over 3 years ago

Feb 2016 Activities

In T10.1

  • Implementation and integration of service and portlet for the FHNManager continued.

In T10.2

  • Investigation on how to move the current Secies Products Discovery service to the new Smartgears framework.

In T10.3

  • Accounting and DocumentStore Lib has been enhanced.
  • New gCube AuthZ Framework, stateful resource and factory pattern support has been integrated in Smartgears 2.0.0.
  • Continued activity of comparison between CouchDB <-> CouchBase <-> MongoDB. This Activity is also useful for new IS.

#24 Updated by Gabriele Giammatteo over 3 years ago

Mar 2016 Activities

In T10.1:
- FHNManager implementation continued. Integration tests between the service (ENG) and the portlet (CNR) have been carried out

In T10.2:
- CNR started an analysis of CKAN platform as data catalogue: a mapping between D4science entities and CKAN entities have been realized and a service for CKAN integration in the D4Science infrastructures have been developed (#3063).

In T10.3:
- Tests on OrientDB have been carried out to check the its performance for storing new Information System resource profiles. This technology has been selected because of its data-model and its scalability, consistency, availability characteristics.
- Accounting and Document-Store Lib has been enhanced and bug fixed
- New gCube AuthZ Framework has been enhanced

#25 Updated by Gabriele Giammatteo over 3 years ago

  • Start date changed from Sep 01, 2015 to Mar 07, 2016
  • Due date changed from Jan 31, 2018 to May 09, 2016

due to changes in a related task

#26 Updated by Gabriele Giammatteo over 3 years ago

Apr 2016 Activities

In T10.1:
- FHNManager service has been released in gCube 3.11.0, documentation available at https://wiki.gcube-system.org/gcube/FHNManager

In T10.2:
- In this period CNR Harvested the EUDAT data catalogue (see: http://b2find.eudat.eu/dataset) using ckan-harvesting plugin.

In T10.3:
- Enhancements have been released for the accounting library and the document-store-lib

#27 Updated by Luca Frosini over 3 years ago

  • Due date changed from Mar 11, 2016 to May 09, 2016

due to changes in a related task

#28 Updated by Gabriele Giammatteo over 3 years ago

May 2016 Activities

  • Integration of FHNManager service and gCube Information System started (will be released in gCube 4)
  • tests on OrientDB for the IS purpose continued. New IS development has been started.
  • Accounting and Document-Store Lib has been enhanced and bug fixed in gCube 3.11.0 release
  • Deliverable D10.1 has been produced by the participants of the WP. Documentation of the various components included in the deliverable have been updated and reviewed (or created if they did not existed).

#29 Updated by Luca Frosini about 3 years ago

  • Due date changed from Apr 30, 2016 to Sep 30, 2016

due to changes in a related task

#30 Updated by Gabriele Giammatteo about 3 years ago

June 2016 Activities

In T10.1:
- integration of FHNManager service with gCube Information System started (will be released in gCube 4)

In T10.2:
- OSCAR files have been imported on Thredds

In T10.3:
- IS developments has been continued refs #3934 #3936 #4023
- Accounting and Document-Store Lib has been ported to CouchBase refs #1665 #4240 #4481 #4254 #4472 #4137 #4318
- AuthZ 2 Framework has been improved and deployed #4141

#31 Updated by Nunzio Andrea Galante about 3 years ago

Jul/Aug 2016 Activities

In T10.1:
- Integration of FHNManager service with gCube Information System has been finalized (gCube 4.0 release)
- Occopus-FHNManager integration development started

In T10.2:
- New products have been imported in the data catalogue

In T10.3:
- First version of IS has been released
- New version of Accounting and Document-Store Lib has been released and deployed on production
- AuthZ 2 Framework has been improved
- Ckan Connector has been released

#32 Updated by Luca Frosini almost 3 years ago

  • Due date changed from Sep 30, 2016 to Dec 31, 2016

due to changes in a related task

#33 Updated by Nunzio Andrea Galante almost 3 years ago

Sept 2016 Activities

In T10.1:

  • Bugs concerning the FHN-Manager fixed
  • Integration between the service and the IS finalized
  • Integration between the service and Occopus finalized
  • Customized the service in order to be used on d4science.org VO

In T10.2:

  • Re-harvesting of FAO Geonetwork products added (#4879)

In T10.3:

#34 Updated by Nunzio Andrea Galante almost 3 years ago

Oct 2016 Activities

In T10.1:

  • Released and deployed the new version of FHNManager Service
    -- first OCCOPUS integration available
    -- IS integration finalized
    -- made the service compliant with new AuthService
    -- made the service compliant with SmartGear 2.0

  • Released the new version of the FHN connector

  • Released the new version of the FHN API

  • Releaded the new version of the FHN-client library

  • Configured environment for the exploitation of new d4science.org resources (EGI)

  • Slide/demo preparation for 3rd TCOM

In T10.2:

  • re-harvesting of modified Metadata for layers generated by Aquamaps (#5625)

In T10.3:

  • IS-Model, gCube Model and Resource Registry and Resource Registry Client and Publisher have been enhanced an bug fixed.
  • OrientDB distrubuted server performance and quality has been tested refs #3477
  • OrientDB and Resource Registry deployment configuration (by using provisioning) has been created and improved taking in account the performance tests #4984 #5568 #5584 #5608

During release period improvements and bug fixes was made on:

#35 Updated by Nunzio Andrea Galante almost 3 years ago

Nov 2016 Activities

WP cross task activities:

  • WP10 Annual Technical Report edited
  • Deliverable 10.1 revisited

In T10.1:

  • FHNManager Service improvements performed:
    -- Added cache mechanism for enriching resource template with informations as cpu and memory
    -- Fixed bug concerning the provider returning information exception

  • Integration with Occopus continued:
    -- On going the development of a mechanism making the nodes created with Occopus uniform with the ones created with the service
    -- Data model modified

  • EGI related activities:
    -- Modified the service in order to allow the creation of VMs based on the new token contextualization with relation to latest Virtual Appliances
    -- Supported CNR for testing new DataMiner and SmartExecutor Virtual Appliances according to the exploitation of new d4science.org resources
    -- Global Grid User Support - GGUS interaction

In T10.2:

  • Interaction with Ocean Biogeographic Information System through REST API in progress

In T10.3:

  • New Information System Activity has been continued:
    -- IS-Model, gCube Model and Resource Registry and Resource Registry Client and Publisher have been enhanced an bug fixed.
    -- OrientDB and Resource Registry deployment configuration (by using provisioning) has improved taking in account use case, partitioning issues.
    -- OrientDB has been upgraded to 2.2.13 which contains bug fixes related to bugs opened to OrientDB team.

  • During release period improvements and bug fixes was made on:
    -- Accounting and Document-Store Lib and related User Interface
    -- AuthZ 2 Framework and related User Interface
    -- Ckan Connector
    -- 3M Connector

#36 Updated by Luca Frosini over 2 years ago

  • Due date changed from Dec 31, 2016 to Feb 28, 2017

due to changes in a related task

#37 Updated by Luca Frosini over 2 years ago

  • Due date changed from Feb 28, 2017 to Jan 31, 2017

due to changes in a related task

#38 Updated by Nunzio Andrea Galante over 2 years ago

Dec 2016 / Jan 2017 Activities

WP cross task activities:

  • WP10 Periodic Technical Report finalized
  • Deliverable 10.1 finalized
  • Milestone MS19 finalized

In T10.1:

  • FHNManager Service improvements performed and bugs fixed

  • Cross BB/EGI activities:
    -- Tested new released Virtual Appliances according to the exploitation of d4science.org resources
    -- Configured resources for all the available providers both at development, preproduction, and production level
    -- Operation support towards GGUS system

In T10.2:

  • Obis plugin for Species Products Discovery have been modified to interact with the new obis webservice
  • Created a script for patching GRSF-Admin's stock records
  • Finished the porting of Species Product Discovery to Smartgears

In T10.3:

  • New Information System Activity has been continued:
    -- IS-Model, gCube Model and Resource Registry and Resource Registry Client and Publisher have been enhanced an bug fixed.
    -- Added publication on new Resource Registry in Smartgears
    -- OrientDB has been upgraded to 2.2.14 in December and to 2.2.15 in January containing some fixed requested by gCube Team

  • During release period support, improvements and bug fixes was made on:
    -- Accounting and Document-Store Lib and related User Interface
    -- AuthZ 2 Framework and related User Interface

  • Developed a web service in order to Enable third-party applications to obtain limited access to D4Science services on behalf of a resource owner via OAuth 2.0

#39 Updated by Nunzio Andrea Galante over 2 years ago

Feb 2017 Activities

In T10.1:

  • FHNManager Service bug fixed
  • New version of FHN portlet has been implemented and it is available in devnext
  • Tested and verified the new DNS service offered by EGI for fedcloud sites
  • Benchmarking of fedcloud computes performed #6581
  • EGI exploitation: four providers are currently offering computational resources

    -- IISAS: 20 cores
    -- GoeGrid: 32 cores
    -- UPV: 8 cores
    -- Recas: 36 cores

  • Cesga site is not working correctly --> GGUS ongoing ticket, waiting for response

Please refer to attached file for a more detailed report about the integration results (#698).

In T10.2:

  • GRSF metadata have been republished on the data catalogue (#7143)

In T10.3:

  • New Information System Activity has been continued
    -- IS-Model, gCube Model, Resource Registry, Resource Registry Client and Resource Registry Publisher have been enhanced an bug fixed. #7227 #7229 #7230 #7231 #7232 #7233 #7234 #7243
    -- OrientDB has been upgraded to 2.2.16 and then to 2.2.17. These versions contains some fixed requested by gCube Team.

  • Improvements and bug fixes was made on:
    -- SmartExecutor #6780 #6781 #6783 #6784
    -- Accounting and Document-Store Lib and related User Interface #6587 #6588 #6596 #6600 #6602 #6603 #6762 #6763 #6980 #7123
    -- AuthZ 2 Framework and related User Interface #6802

#40 Updated by Luca Frosini over 2 years ago

  • Due date changed from Jan 31, 2017 to Apr 30, 2017

due to changes in a related task

#41 Updated by Nunzio Andrea Galante over 2 years ago

Mar 2017 Activities

In T10.1:

In T10.2:

Developed a script for the generation of a Taxa Authority File for WoRMS using the Species Product Discovery service.

In T10.3:

#42 Updated by Nunzio Andrea Galante over 2 years ago

Apr 2017 Activities

In T10.1:

  • Continued the activity of integration with EGI FedCloud sites (#6581)
    -- IISAS FedCloud: five instances to be exploited in production since May (total amount of 20 cores and 50 GB RAM) (#6584)
    -- Recas Bari: 4 instances to be exploited in production since May (total amount of 20 cores and and 48 GB RAM) (#6585)
    -- GoeGrid: 2 instances to be exploited in production since May (total amount of 32 cores and 64 GB RAM) (#6583)
    -- UPV-GryCAP: 1 single instance equipped with 10 cores and 4 GB RAM could be expoit in production since May, upon uploaded the new version of DataMiner able to fix the "Disabling reverse NAT Service for outgoing traffic" issue (#6586)
    -- Cesga (#6582): So far, no resources will be exploited in production but many issues have been solved in these last days, just some issues pending.

  • EGI overall report available at: https://docs.google.com/spreadsheets/d/1l6uHPiSsRL443i4nxRpZ7SJSCMDR39CtR_JTTYF_fs0/edit#gid=707024718

In T10.2:

  • Third-parties community tools integrated according to the needs of WP5-WP8 :
    • Rshiny
    • OpenCPU

In T10.3:

  • New Information System Activity has been continued
    • IS-Model, gCube Model, Resource Registry, Resource Registry Client and Resource Registry Publisher have been enhanced an bug fixed.
    • Monitoring Portlet has been ported to to be used with new IS #7886
    • OrientDB has been upgraded to 2.2.18 and then to 2.2.19 containing some fixed requested by gCube Team.
  • SmartExecutor #7948 #7949
  • Quota Management, Accounting and Document-Store Lib and related User Interface #7258 #8038 #8268
  • AuthZ 2 Framework and related User Interface and Functional Tests #7952 #7937
  • Smartgears has been enhanced to support new Resource Registry publishing #7980 #7979

#43 Updated by Luca Frosini over 2 years ago

  • Due date changed from Apr 30, 2017 to Jun 30, 2017

due to changes in a related task

#44 Updated by Nunzio Andrea Galante over 2 years ago

May 2017 Activities

In T10.1:

  • Continued the activity of integration with EGI FedCloud sites (#6581)

    -- new images uploaded in Appdb (version 1.9 of SmartExecutor and version 1.7 of DataMiner)
    -- new images deployed on the sites (many issues found, actually solved by means of GGUS ticketing)
    -- carried out the packaging of new resources in the IS (so far at Development level)
    -- new contextualization script available for all the environments
    -- new proxy backend to configure for EGI instances
    -- some technical issue observed in the contextualization script of the new images, currently manual workaround adopted
    -- any resources are being exploited in production since may (devnext scope) and should have to be candidates for the data accounting; in particular:

    • IISAS FedCloud: five instances to be exploited in production since May (total amount of 20 cores and 50 GB RAM) (#6584) --> actually two VMs are working: vm125.ui.savba.sk and vm123.ui.savba.sk
    • Recas Bari: 4 instances to be exploited in production since May (total amount of 20 cores and and 48 GB RAM) (#6585) --> actually two VMs are working: cloud-90-147-102-44.cloud.ba.infn.it and cloud-90-147-102-20.cloud.ba.infn.it
    • GoeGrid: 2 instances to be exploited in production since May (total amount of 32 cores and 64 GB RAM) (#6583); actually just one is running: c096-068.cloud.gwdg.de
    • UPV-GryCAP: 1 single instance equipped with 10 cores and 4 GB RAM could be expoit in production since May --> the new version of DataMiner seems to be not able to fix the "Disabling reverse NAT Service for outgoing traffic" issue as expected(#6586)
    • Cesga (#6582): So far, no resources will be exploited in production --> new version of the image seems to be not able to manage the hostname resolution as expected

In T10.2:

  • GRSF Publisher / Catalogue - support for record time dependent and independent fields (#8719)

In T10.3:

All reported activities contains migration to java 8 effort.

#45 Updated by Nunzio Andrea Galante about 2 years ago

Jun 2017 Activities

In T10.1:

  • Continued the activity of integration with EGI FedCloud sites (#6581)
    -- new contextualization script available for all the environments
    -- strong test activities performed
    -- many issues found, actually solved by means of GGUS ticketing
    -- any resources are being exploited in production since may (devnext scope) and we are collecting data accounting for three of the five sites providing resources to d4science.org VO (total amount of: 14 active instances, 64 cores and 152 GB RAM).

    • IISAS FedCloud: six instances exploited in production since June (total amount of 24 cores and 48 GB RAM) (#6584) --> actually six VMs are working: vm99.ui.savba.sk, vm100.ui.savba.sk, vm108.ui.savba.sk, vm109.ui.savba.sk, vm123.ui.savba.sk, vm125.ui.savba.sk
    • Recas Bari: 3 instances exploited in production since June (total amount of 20 cores and and 40 GB RAM) (#6585) --> actually three VMs are working: cloud-90-147-170-49.cloud.ba.infn.it, cloud-90-147-170-20.cloud.ba.infn.it, cloud-90-147-102-44.cloud.ba.infn.it
    • GoeGrid: 5 instances exploited in production since June (total amount of 20 cores and 64 GB RAM) (#6583) --> actually five VMs are working: c096-068.cloud.gwdg.de, c096-075.cloud.gwdg.de, c096-076.cloud.gwdg.de, c096-077.cloud.gwdg.de, c096-079.cloud.gwdg.de
    • UPV-GryCAP: 1 single instance equipped with 10 cores and 4 GB RAM could be expoit in production since May --> the new version of DataMiner seems to be not able to fix the "Disabling reverse NAT Service for outgoing traffic" issue as expected(#6586)
    • Cesga (#6582): So far, no resources will be exploited in production --> new version of the image seems to be not able to manage the hostname resolution as expected

In T10.2:

  • The Oscar merger/importer integrated as a scheduled job for the smart executor. Deployed in the dev infrastrcuture and uploading the merged dataset to the dev thredds server. (#8713)

In T10.3:

#46 Updated by Nunzio Andrea Galante about 2 years ago

Jul/Aug 2017 Activities

  • Finalized the milestone document MS25 - BlueBRIDGE Resources Federation Facilities: IV Major Release (#706).

In T10.1:

  • Continued the activity of integration with EGI FedCloud sites (#6581)
    • Some issues found, actually all solved by means of GGUS ticketing
    • New Virtual Appliance Version to be uploaded once will be solved the following: https://ggus.eu/index.php?mode=ticket_info&ticket_id=129803
    • Except CESGA provider, resources from IISAS FedCloud, Recas Bari, GOEGrid and UPV are currently being exploited in production.

In T10.2:

  • Started the analysis of the dataset from Copernicus, in order to realize some automatic merger/importer as a scheduled job for the smart executor towards the pubblication in the gCube Infrastructure
  • EMODnet GeoNetwork harvesting (#9494)
  • ICES GeoNetwork harvesting (#9286)

In T10.3:

  • New Information System Activity has been continued
    • IS-Model, gCube Model, Resource Registry, Resource Registry Client and Resource Registry Publisher have been enhanced an bug fixed. #9221 #9021
  • SmartExecutor #8523 #8525
  • Quota Management, Accounting and Document-Store Lib and related User Interface
  • Smartgears #9118 #9493 #9309

#47 Updated by Luca Frosini about 2 years ago

  • Due date changed from Jun 30, 2017 to Dec 31, 2017

due to changes in a related task

#48 Updated by Nunzio Andrea Galante almost 2 years ago

Sep 2017 Activities

In T10.1:

  • Continued the exploitation of the EGIFedCLoud resources
  • Added Features to FHNManager Service and bug fixing (#9726)(#9863)(#9703)(#4718)
  • Released the new versions of the following components:
    • fhn-occi-connector.1-1-2 (#9710)
    • fhn-manager-api.1-2-2 (#9709)
    • fhn-manager-service.1-2-3 (#9711)
  • Support to CNR to make ansible-compliant the installation and configuration of the FHNManager Service

In T10.2:

  • ICES GeoNetwork data harvesting (#9286)
  • EMODnet Organisation in CKAN BlueBRIDGE Catalogue heve been created (#9495)
  • On going the realization of the automatic merger/importer for the Copernicus dataset

In T10.3:

  • New Information System Activity has been continued
    • IS-Model, gCube Model, Resource Registry, Resource Registry Client and Resource Registry Publisher have been enhanced an bug fixed. #9772
  • SmartExecutor has been bug fixed.
  • Quota Management, Accounting and Document-Store Lib and related User Interface. #9032 #9033 #9034 #9035 #9036 #9612 #9617 #9629 #9646 #9647 #9649
    • Accounting Aggregator Plugin has been enhanced and deployed on production. #9419 #9556 Actually With DAILY and MONTHLY aggregation the number of ServiceUsageRecords documents from 26.125.206 were reduced to 3.963.402. #8778 #9648 #9656
  • Smartgears #9833

#49 Updated by Paolo Fabriani almost 2 years ago

October 2017 Activities

In T10.1:

  • Continued the exploitation of the EGIFedCLoud resources

In T10.2

  • Merger/importer for the Copernicus dataset: a client for the Motu server has been implemented supporting CAS authentication and request splitting to fulfill server policies.

In T10.3

#50 Updated by Paolo Fabriani almost 2 years ago

November 2017 Activities

In T10.2

  • Merger/importer for Copernicus CMEMS datasets (#10329):
    • definition of a new service and associated GUI to allow users to autonomously browse CMEMS datasets and schedule import jobs (#10331)
    • continued the implementation of the CMEMS/Motu client (#10337):
      • support for splitting request over variables (besides time, already implemented);
      • operations to build a catalogue of datasets published by a Motu server;
      • merge of results of split requests (both along time and variables);
      • started implementation of a client for the service behind the CMEMS portal; to complement information retrieved from Motu servers.

In T10.3

  • New Information System Activity has been continued
    • IS-Model, gCube Model, Resource Registry, Resource Registry Client and Resource Registry Publisher have been enhanced an bug fixed.
    • Resource Registry service has been re-factored to improve performances, code readability and future evolution. Moreover an improved context management has been implemented.
    • Development of New Resource Registry Context Client and Resource Registry Schema Client has been started
  • New versoin of SmartExecutor has been released #10113
  • Quota Management, Accounting and Document-Store Lib and related User Interface. #10376 #10110 #10111
  • Smartgears #9599 #9835 #10269

#51 Updated by Luca Frosini almost 2 years ago

  • Due date changed from Dec 31, 2017 to Jun 01, 2017

due to changes in a related task

#52 Updated by Paolo Fabriani over 1 year ago

December 2017 Activities

In T10.2

Importer for Copernicus CMEMS datasets (#10329):

  • CMEMS Client: completed implementation of a client of the service behind the CMEMS portal to quickly search and browse all CMEMS datasets. This operation also allows to know the Motu server publishing each dataset (#10332).

  • SmartExecutor Plugin

    • the plugin, given an import request, is now able to 1) retrieve the current import status on Thredds; 2) compute the missing portion of the dataset; 3) download all updates from CMEMS; 4) upload the updates to Thredds 5) update the virtual dataset descriptor (.ncml). (#10339)
    • Started implementation of a client library to interact with the smart executor (only for scheduling and monitoring tasks. The API to get the list of scheduled tasks is being implemented at #10780). (#10333)
  • As an complement/alternative to the GUI, implemented a command-line interface to search, browse datasets and list, schedule, un-schedule, monitor import tasks. (#10333)

In T10.3

Accounting and Document-Store Lib and related User Interface. #10146 #10810 #10715 #10804 #10646 #10142 #10677 #10742

  • Exploited new Accounting Dashboard for BlueBRIDGE contexts.

#53 Updated by Paolo Fabriani over 1 year ago

January 2017 Activities

In T10.2

Importer for Copernicus CMEMS datasets (#10329):

  • Dataset Importer Service - design and development of a SmartGears service exposing dataset browsing facilities, scheduling import tasks and collect and expose report (logs) of import task executions. The service also exposes the status and progress of each execution. The main client of the service is the dataset importer GUI.

    • Based on, and exposing features of the CMEMS and Motu clients already developed.
    • Added new operations to publish/retrieve execution status and logs.
    • Validation of import request parameters against CMEMS/Motu servers to early discover execution-time errors.
    • Security.
  • SmartExecutorPlugin

    • interaction with the DatasetImporterService to publish execution logs, status and progress.
  • Dataset Importer (GUI) - a portlet to allow users to easily browse datasets, manage (schedule, stop, monitor) import tasks, view reports.

    • Design completed
    • Browse and view of products, datasets and tasks is working.
    • Ongoing: widget to submit a new import task
    • GWT-Bootstrap adopted as UI technology

In T10.3

SmartExecutor

  • REST interface has been added to SmartExecutor #5109

#54 Updated by Pasquale Pagano over 1 year ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF