Support #16234

Storage client performance decline

Added by Fabio Sinibaldi 11 months ago. Updated 9 months ago.

Status:ClosedStart date:Mar 06, 2019
Priority:UrgentDue date:
Assignee:Tommaso Piccioli% Done:

100%

Category:content-management
Milestones:
Duration:

Description

I'm trying to figure out the cause of a drastic performance decline in perform-service interaction with storage.

Please note that the following behavior has only been found in perform.dev.d4science.org node, and starting from the 4/3/2019 at about 11:30 am.

At the present situation, the following code :
~~~
log.debug("Uploading local file "+source.getAbsolutePath());
String id=client.put(true).LFile(new FileInputStream(source)).RFile(UUID.randomUUID().toString());
log.debug("File uploaded. ID : "+id);
~~~

produces the following log lines :
~~~
2019-03-06 12:10:44,150 [catalina-exec-3] DEBUG StorageUtils: Uploading local file /home/gcube/tomcat/tmp/csv_out4642122027192330603.csv
2019-03-06 12:10:59,296 [catalina-exec-3] DEBUG StorageUtils: File uploaded. ID : 5c7faab4a98bdc6392e2e3e2
~~~

As you can see from the log timestamps, it seems that the upload took about 15secs (the csv file in this case is just 40 bytes).

Since this happens at every similar interaction, and apparently only on that VM, I tend to assume an I/O issue is the cause.

Maybe something happened the 4th of march?

History

#1 Updated by Roberto Cirillo 11 months ago

This time is very very high. From my machine I have 0.2" uploading a small file. I see that your file has been uploaded in the volatile area, in dev environment.
But on mongoDb nothing has changed in these days. @tommaso.piccioli@isti.cnr.it do you think is it possible a VM problem here?

#2 Updated by Tommaso Piccioli 11 months ago

I don't see any issue on perform.dev.d4science.org at the moment

#3 Updated by Roberto Cirillo 11 months ago

I've performed some tests from my machine and from perform-dev VM using a small csv file.

This is the result from my machine:

rcirillo@rcirillo-cnr:~/Documents/INFRASTRUCTURE/Storage/Storage-CLI-Examples/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 334 ms. Size in byte: 58
rcirillo@rcirillo-cnr:~/Documents/INFRASTRUCTURE/Storage/Storage-CLI-Examples/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 4355 ms. Size in byte: 58
rcirillo@rcirillo-cnr:~/Documents/INFRASTRUCTURE/Storage/Storage-CLI-Examples/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 445 ms. Size in byte: 58
rcirillo@rcirillo-cnr:~/Documents/INFRASTRUCTURE/Storage/Storage-CLI-Examples/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 1632 ms. Size in byte: 58
rcirillo@rcirillo-cnr:~/Documents/INFRASTRUCTURE/Storage/Storage-CLI-Examples/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 1184 ms. Size in byte: 58

This is the result from perform VM:

gcube@perform:~/storageTest$ cd Storage-CLI-smallfiles/
gcube@perform:~/storageTest/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 45463 ms. Size in byte: 58
gcube@perform:~/storageTest/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 47854 ms. Size in byte: 58
gcube@perform:~/storageTest/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 45456 ms. Size in byte: 58
gcube@perform:~/storageTest/Storage-CLI-smallfiles$ ./startUpload.sh 
Time upload (ms): 45915 ms. Size in byte: 58

Seems that from perform VM the time is very very high compared to the time from my machine.

#4 Updated by Roberto Cirillo 11 months ago

  • Assignee changed from Roberto Cirillo to Tommaso Piccioli
  • Status changed from New to Feedback

#6 Updated by Roberto Cirillo 11 months ago

The traceroute output from perform-dev is the following:

first attempt:

gcube@perform:~$ traceroute mongo-d-vol.d4science.org
traceroute to mongo-d-vol.d4science.org (146.48.122.94), 30 hops max, 60 byte packets
 1  mongo-d-vol.d4science.org (146.48.122.94)  0.112 ms !X  0.093 ms !X  0.071 ms !X

second attempt:

gcube@perform:~$ traceroute mongo-d-vol.d4science.org
traceroute to mongo-d-vol.d4science.org (146.48.122.94), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * mongo-d-vol.d4science.org (146.48.122.94)  0.132 ms !X  0.118 ms !X

I don't know if it is a normal behavior

#8 Updated by Roberto Cirillo 9 months ago

  • Priority changed from Normal to Urgent

#10 Updated by Tommaso Piccioli 9 months ago

  • % Done changed from 0 to 100
  • Status changed from Feedback to Closed

Problem solved taking "perform" as hostname instead of "perform.dev"
(FQDN is still perform.dev.d4science.org)

The problem arose last month due to this recent inclusion in top-level domain names:
https://en.wikipedia.org/wiki/.dev

#11 Updated by Andrea Dell'Amico 9 months ago

Tommaso Piccioli wrote:

Problem solved taking "perform" as hostname instead of "perform.dev"
(FQDN is still perform.dev.d4science.org)

The problem arose last month due to this recent inclusion in top-level domain names:
https://en.wikipedia.org/wiki/.dev

Nice spot!

Also available in: Atom PDF