Feature #1442

BlueBRIDGE - Project WP #686: WP9 - VRE Commons Development [Months: 1-30]

BlueBRIDGE - Project Task #687: T9.1 Data Storage, Discovery and Access [Months: 1-29]

Integrate OAI-PMH directly to harvesting/indexing mechanism

Added by George Kakaletris about 4 years ago. Updated about 3 years ago.

Status:ClosedStart date:Nov 20, 2015
Priority:NormalDue date:
Assignee:Panagiota Koltsida% Done:

100%

Category:index-management
Sprint:zz - Search and Index Enhancement
Milestones:
Duration:

Related issues

Related to BlueBRIDGE - Project Task #700: T10.3 Federated Resources Management [Months: 1-29] Closed Mar 07, 2016 Apr 30, 2016

History

#1 Updated by George Kakaletris about 4 years ago

  • Related to Feature #1440: Remove index ingestion dependency to TreeManager added

#2 Updated by Pasquale Pagano about 4 years ago

We are thinking to add a new service to the enabling layer. It will be a sort of meta-scheduler. Currently it supports typical invocation of repetitive tasks with the same cron capabilities. This will allow to schedule the tasks at level of different VREs. Harvesting and frequency of harvesting could then be managed by this new service. It just needs a service to call periodically. In this sense you may avoid to implement everything about scheduling in your OAI-PMH harvesting.

#3 Updated by Massimiliano Assante almost 4 years ago

I very welcome the fact that you "linked" this ticket to the work package, perhaps it would be better to "attach" these activities to WP9 Tasks (T9.1, T9.2 or T9.3) instead of the WP directly (Set as Parent task, or related, the T9.1, T9.2 or T9.3 and not WP9)

Is this sound for you?

#4 Updated by Panagiota Koltsida almost 4 years ago

  • Parent task changed from #686 to #687

#5 Updated by Panagiota Koltsida almost 4 years ago

  • Status changed from New to In Progress

#6 Updated by Massimiliano Assante almost 4 years ago

I'm removing the parent task of this ticket as it was wrongly attached as subtask of the WP 9 ticket. The correct way is to put this ticket as related to the corresponding Project Task (T9.1 in this case)

also, @p.koltsida@di.uoa.gr could you please provide us with an update on the status of this activity? It is in progress since 2 months, no further information have been added. It would be beneficial if you could set an estimation of the due date for this activity.

#7 Updated by Massimiliano Assante almost 4 years ago

  • Parent task deleted (#687)

#8 Updated by Panagiota Koltsida almost 4 years ago

  • % Done changed from 0 to 50

Currently we have almost finished the implementation of a generic OAI-PMH harvester service and it remains to directly store the harvested records to the available index by directly interacting with it.
The OAI-PMH harvester can harvest OAI-PMH services that are registered to it and the harvesting is done either periodically (per URI if set) or explicitly when requested.

#9 Updated by Panagiota Koltsida over 3 years ago

  • Related to Project Task #700: T10.3 Federated Resources Management [Months: 1-29] added

#10 Updated by Panagiota Koltsida over 3 years ago

  • Related to Project Task #687: T9.1 Data Storage, Discovery and Access [Months: 1-29] added

#11 Updated by Massimiliano Assante over 3 years ago

  • Related to deleted (Project Task #687: T9.1 Data Storage, Discovery and Access [Months: 1-29])

#12 Updated by Massimiliano Assante over 3 years ago

  • Parent task set to #687

#13 Updated by Massimiliano Assante over 3 years ago

  • Related to deleted (Feature #1440: Remove index ingestion dependency to TreeManager )

#14 Updated by Panagiota Koltsida over 3 years ago

  • % Done changed from 50 to 90

An OAI-PMH harvester has been developed and will be released in 4.1 gcube release. The new OAI-PMH harvests and directly puts the records into the elastic search engine

#15 Updated by Panagiota Koltsida over 3 years ago

  • % Done changed from 90 to 100

#16 Updated by Panagiota Koltsida about 3 years ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF