Feature #6175

Accounting Analytics - Analysis to speed up the calculation of the top

Added by Alessandro Pieve almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:Dec 12, 2016
Priority:NormalDue date:
Assignee:Alessandro Pieve% Done:

100%

Category:accounting
Sprint:zz - UnSprintable
Milestones:
Duration:

Description

Analysis to modify the calculation of the top switching from a query approach to some other technology, for example a map-reduce.


Related issues

Related to gCube - Release #6596: org.gcube.accounting.accounting-analytics.2-4-0 Released Jan 25, 2017
Related to gCube - Release #6600: org.gcube.accounting.accounting-analytics-persistence-cou... Released Jan 25, 2017

History

#1 Updated by Luca Frosini almost 3 years ago

  • Estimated time set to 6.00
  • Status changed from New to In Progress

Friday talking with Alessandro I had an Idea. Today morning We tested such idea together and seem working.

#2 Updated by Alessandro Pieve almost 3 years ago

The solutions found do not seem to work in production.
The system has too much data and fails to stabilize (loop for create an index).
I'm analyzing possible improvements.

Error log:
mapreduce_errors:error,2016-12-19T10:21:29.397+01:00,couchdb_ns_1@127.0.0.1:<0.21371.208>:couch_set_view_util:log_port_error:813]Set view accounting_service, main group _design/dev_topUsage, received error from index builder: Error building index for view calledMethod, reason: (view 5) reduction too large, ret = -13

#3 Updated by Alessandro Pieve almost 3 years ago

  • % Done changed from 0 to 40

#4 Updated by Alessandro Pieve almost 3 years ago

  • % Done changed from 40 to 70

Optimized further, but always with a problem regarding the top for calledMethod (reduction too large)
The library was updated (accounting-analytics-persistence-couchbase 1.3.0-SNAPSHOT )and loaded in "https://dev4.d4science.org/group/nextnext/accounting-manager"

#5 Updated by Alessandro Pieve almost 3 years ago

In testing, i found a problem when calculate top with multiple filter , solved

#6 Updated by Alessandro Pieve almost 3 years ago

  • % Done changed from 70 to 100
  • Status changed from In Progress to Closed

#7 Updated by Alessandro Pieve almost 3 years ago

  • Related to Release #6596: org.gcube.accounting.accounting-analytics.2-4-0 added

#8 Updated by Alessandro Pieve almost 3 years ago

  • Related to Release #6600: org.gcube.accounting.accounting-analytics-persistence-couchbase.1-3-0 added

Also available in: Atom PDF