Bug #11013

Ansible: problems after the upgrade of the smartgears container

Added by Roberto Cirillo almost 2 years ago. Updated almost 2 years ago.

Status:ClosedStart date:Jan 23, 2018
Priority:UrgentDue date:
Assignee:Andrea Dell'Amico% Done:

100%

Category:tools
Sprint:zz - UnSprintable
Milestones:
Duration:

Description

I've tried to upgrade some services in preproduction environment using the following playbook:

smartgears-node-upgrade.yml

The two nodes are: "catalogue-ws-t" and "socialnetworking-t" both running with (oracle-java-8 jdk).
After the upgrade the container doesn't start. If I try to start it manually I've the following exception:

gcube@catalogue-ws-t:~$ ./startContainer.sh 
 * Starting Tomcat servlet engine tomcat-instance-9000      

                                                                                                                                                        start-stop-daemon: unable to stat /usr/lib/jvm/java-8-openjdk-amd64/bin/java (No such file or directory)

The problem is that the file /etc/default/tomcat-instance-9000 is pointing to a not existing open-jdk as reported here:

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

The same playbook was runnning successfully yesterday. I guess this bug is due to the last commit on git.

Please could you check asap?


Related issues

Related to gCube - Bug #11020: CXF versions < 3.2.2 do not work with the Oracle JDK buil... Closed Jan 24, 2018

History

#1 Updated by Andrea Dell'Amico almost 2 years ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

I just pushed a fix. I hope it's not permanent because it prevents to switch between the Oracle JDK and Openjdk without setting an additional variable.

#2 Updated by Roberto Cirillo almost 2 years ago

  • Status changed from Closed to Feedback

I reopened this ticket because now the tomcat script starts correctly but the container not, with the following exception:

Caused by: java.lang.UnsupportedOperationException: Entity References are not allowed in SOAP documents
        at com.sun.xml.internal.messaging.saaj.soap.SOAPDocumentImpl.createEntityReference(SOAPDocumentImpl.java:148)
        at com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.createEntityReference(SOAPPartImpl.java:437)
        at com.sun.xml.internal.ws.api.message.saaj.SaajStaxWriter.writeEntityRef(SaajStaxWriter.java:245)
        at com.sun.xml.internal.bind.v2.runtime.output.XMLStreamWriterOutput$XmlStreamOutWriterAdapter.writeEntityRef(XMLStreamWriterOutput.java:277)
        at com.sun.xml.internal.bind.v2.runtime.output.XMLStreamWriterOutput$NewLineEscapeHandler.escape(XMLStreamWriterOutput.java:242)
        at com.sun.xml.internal.bind.v2.runtime.output.XMLStreamWriterOutput.text(XMLStreamWriterOutput.java:150)
        at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.leafElement(XMLSerializer.java:313)
        at com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$StringImplImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:1036)
        at com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$StringImplImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:1015)
        at com.sun.xml.internal.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.writeLeafElement(TransducedAccessor.java:239)
        at com.sun.xml.internal.bind.v2.runtime.property.SingleElementLeafProperty.serializeBody(SingleElementLeafProperty.java:115)
        at com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:345)
        at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:681)
        at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:256)
        at com.sun.xml.internal.bind.v2.runtime.BridgeImpl.marshal(BridgeImpl.java:79)
        at com.sun.xml.internal.bind.api.Bridge.marshal(Bridge.java:96)
        at com.sun.xml.internal.ws.db.glassfish.BridgeWrapper.marshal(BridgeWrapper.java:177)
        at com.sun.xml.internal.ws.message.jaxb.JAXBMessage.writePayloadTo(JAXBMessage.java:401)
        at com.sun.xml.internal.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:173)
        at com.sun.xml.internal.ws.api.message.MessageWrapper.writeTo(MessageWrapper.java:206)
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:270)
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:197)
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:186)
        at com.sun.xml.internal.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:217)
        at com.sun.xml.internal.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:156)
        at com.sun.xml.internal.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:70)
        at org.gcube.common.clients.stubs.jaxws.handlers.GCoreJAXWSHandler.handleMessage(GCoreJAXWSHandler.java:34)
        ... 51 common frames omitted

After the last playbook run, the jdk was changed in "1.8.0_161" while the containers upgraded in the last days the java version was "1.8.0_151".

I think the problem could be related to the jdk upgrade as reported here: https://issues.apache.org/jira/browse/CXF-7520
Is it possible to restore the old oracle-jdk version "1.8.0_151" on "socialnetworking-t.pre" in order to check if is this the problem?

#3 Updated by Andrea Dell'Amico almost 2 years ago

But this is a completely different kind of problem and it can be only fixed in one of these two ways:

  • Use OpenJDK (where we can stick to a specific version for a while, if it's necessary)
  • set the jdk_pkg_state: to present globally, to avoid jdk upgrades

The second solution is not applicable to new VMs obviously. The 161 build fixes some nasty security bugs, so we should be able to use it ASAP.

#4 Updated by Andrea Dell'Amico almost 2 years ago

Roberto Cirillo wrote:

After the last playbook run, the jdk was changed in "1.8.0_161" while the containers upgraded in the last days the java version was "1.8.0_151".

I think the problem could be related to the jdk upgrade as reported here: https://issues.apache.org/jira/browse/CXF-7520
Is it possible to restore the old oracle-jdk version "1.8.0_151" on "socialnetworking-t.pre" in order to check if is this the problem?

You need to copy the contents of /usr/lib/jvm/java-8-oracle/ from another VM, there's no way to cleanly install an older Oracle JDK.

#5 Updated by Roberto Cirillo almost 2 years ago

Andrea Dell'Amico wrote:

Roberto Cirillo wrote:

After the last playbook run, the jdk was changed in "1.8.0_161" while the containers upgraded in the last days the java version was "1.8.0_151".

I think the problem could be related to the jdk upgrade as reported here: https://issues.apache.org/jira/browse/CXF-7520
Is it possible to restore the old oracle-jdk version "1.8.0_151" on "socialnetworking-t.pre" in order to check if is this the problem?

You need to copy the contents of /usr/lib/jvm/java-8-oracle/ from another VM, there's no way to cleanly install an older Oracle JDK.

OK I would try it. Is it possible to do it by rsync tool? For instance, the "sdi-t.pre.d4science.org" node has the "1.8.0_151" java version.

I need to understand if our smartgears container is compatible with oracle-jdk "1.8.0_161" in order to avoid, eventually, this upgrade on production nodes.

#6 Updated by Andrea Dell'Amico almost 2 years ago

Yes, you can log into sdi-t.pre.d4science.org and run

rsync -a --delete /usr/lib/jvm/java-8-oracle/ catalogue-ws-t.d4science.org:/usr/lib/jvm/java-8-oracle

(not tested, verify that files are written on the right place on the target VM)

#7 Updated by Andrea Dell'Amico almost 2 years ago

  • Related to Bug #11020: CXF versions < 3.2.2 do not work with the Oracle JDK build 161 added

#8 Updated by Andrea Dell'Amico almost 2 years ago

  • Status changed from Feedback to Closed

We have a proper ticket now, I'm closing this one.

#9 Updated by Roberto Cirillo almost 2 years ago

ok thanks. I'm going to test it.

#10 Updated by Roberto Cirillo almost 2 years ago

  • Subject changed from Ansible-playbook: bad java configuration on tomcat after smartgears-upgrade-node run to Ansible: problems after the upgrade of the smartgears container

#11 Updated by Lucio Lelii almost 2 years ago

We found how to solve this problem via code.
This is related to some illegal (for SOAP) characters in the request; in java8-build 1.6.1 they removed the possibility to escape them.
Also Jaxb native indentation inserts some invalid chars and now it is disabled in our code.
I have just released 2 libaries: ic-client and common-gcore-resource.
I have tested this solution on socialnetworking-t.

Also available in: Atom PDF