Incident #12894

Update of FSK plugin failed

Added by Lars Valentin 5 months ago. Updated 4 months ago.

Status:ClosedStart date:Nov 16, 2018
Priority:HighDue date:
Assignee:Andrea Dell'Amico% Done:

100%

Category:Other
Infrastructure:Development, Pre-Production, Production
Milestones:
Duration:

Description

Hello Andrea,

there is at the moment no KNIME FSKlab plugin installed in the DataMiner.

I am sure this was caused by us due to an uploading issue to the update repository.

Could you please run Ansible over again. I just fixed the repository.

Thank you!

Lars

d4science-ansible-knime-role.tar.gz - ansible knime role (49 KB) Andrea Dell'Amico, Dec 09, 2018 12:35 PM

History

#1 Updated by Andrea Dell'Amico 5 months ago

  • Status changed from New to In Progress

#2 Updated by Andrea Dell'Amico 5 months ago

  • Assignee changed from Andrea Dell'Amico to Lars Valentin

The installation still fails:

!SESSION 2018-11-16 13:17:40.660 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-11-16 13:17:46.848
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.

The command invoked to install the plugin is:

knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf -installIU de.bund.bfr.knime.fsklab.feature.feature.group

#3 Updated by Lars Valentin 5 months ago

Thank you Andrea for trying. I reported your findings to our developers and they seem to have an idea why it does not work via the commandline this time. It seems to be a missing dependency. As soon as we solved it, I will let you know.

#4 Updated by ahmad swaid 4 months ago

  • Priority changed from Urgent to Normal
  • Assignee changed from Lars Valentin to Andrea Dell'Amico

Hi Andrea,
The problem was in the update site of FSK_Lab which cause problems in the cache of P2 repository of KNIME.
We are investigating the problem so that it will not happen again in the future. So it will be very nice if you provide us with the ansible script or part of it which do the installation of the plugins in KNIME headless.

Best regards
Ahmad

#5 Updated by Andrea Dell'Amico 4 months ago

The daily update is performed by a cron job that runs a shell script:

#!/bin/bash
RETVAL=
RETVAL_WF=
DATE=$( date )
export PATH="/usr/local/bin:$PATH"

if [ ! -d /opt/knime/logs ] ; then
    mkdir /opt/knime/logs
fi

if [ -f "/opt/knime/scripts/lib_knime_fsk_updater" ] ; then
    . /opt/knime/scripts/lib_knime_fsk_updater
else
    logger "knime_fsk_updater: /opt/knime/scripts/lib_knime_fsk_updater file, exiting"
    exit 1
fi

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -uninstallIU "${plugin}" > "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            rm -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} removed successfully"
        else
            logger "knime_fsk_updater: Plugin ${plugin} removal failed"
        fi
    fi
done

RETVAL=

for plugin in ${KNIME_FSK_PLUGINS} ; do
    if [ ! -f "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed" ] ; then
        knime -nosplash -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination "${KNIME_WORKING_PATH}" -repository "${KNIME_PLUGINS_REPOSITORY}" -installIU "${plugin}" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
        RETVAL=$?
        if [ $RETVAL -eq 0 ] ; then
            echo "$DATE" > "${KNIME_WORKING_PATH}/plugins/.${plugin}.installed"
            logger "knime_fsk_updater: Plugin ${plugin} installed successfully"
            knime -nosplash -clean -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="${KNIME_WORKFLOWS_DIR}/simple_FSK_testworkflow" >> "$KNIME_PLUGINS_LOG_FILE" 2>&1
            RETVAL_WF=$?
            if [ $RETVAL_WF -eq 0 ] ; then
                logger "knime_fsk_updater: simple FSK workflow run successfully"
            else
                logger "knime_fsk_updater: simple FSK workflow failed"
                exit $RETVAL_WF
            fi
        else
            logger "knime_fsk_updater: Plugin ${plugin} failed to install"
        fi
    fi
done

chown -R gcube:gcube ${KNIME_WORKING_PATH}/

exit $RETVAL

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"
KNIME_FSK_PLUGINS="de.bund.bfr.knime.fsklab.feature.feature.group "
KNIME_WORKING_PATH="/opt/knime/knime_distribution"
KNIME_PLUGINS_LOG_FILE="/opt/knime/logs/knime_fsk_plugins.log"
KNIME_WORKFLOWS_DIR=/opt/knime/knime_workflows

#6 Updated by Andrea Dell'Amico 4 months ago

Here is the ansible knime role, anyway. I've not reworked it to supporto multiple knime versions yet.

#7 Updated by Lars Valentin 4 months ago

Thank you Andrea,

since we have combined our repositories, https://dl.bintray.com/silebat/fsklab_icpmf
does include now https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/. Therefore, https://dl.bintray.com/silebat/pmmlab and http://dl.bintray.com/silebat/internal/ need to be removed from all the scripts.

In addition https://dl.bintray.com/silebat/fsklab_icpmf has dependencies to the KNIME main repository http://update.knime.org/analytics-platform/{{ knime_update_version }} and therefore should be always used in combination. I think this needs to be added to your cron job.

The file /opt/knime/scripts/lib_knime_fsk_updater contains

KNIME_PLUGINS_REPOSITORY="https://dl.bintray.com/silebat/fsklab_icpmf"

Best,
Lars

#8 Updated by Andrea Dell'Amico 4 months ago

  • Status changed from In Progress to Feedback

I changed the playbook and the cron job configuration. The plugin installation still fails:

eclipse.buildId=unknown
java.version=1.8.0_152
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group
Command-line arguments:  -os linux -ws gtk -arch x86_64 -application org.eclipse.equinox.p2.director -profile KNIMEProfile -destination /opt/knime/knime_distribution -repository https://dl.bintray.com/silebat/fsklab_icpmf,http://update.knime.org/analytics-platform/3.6 -installIU de.bund.bfr.knime.fsklab.feature.feature.group

!ENTRY org.eclipse.equinox.p2.director.app 4 0 2018-12-10 12:09:50.891
!MESSAGE The installable unit de.bund.bfr.knime.fsklab.feature.feature.group has not been found.

#9 Updated by Lars Valentin 4 months ago

  • % Done changed from 100 to 20
  • Priority changed from Normal to High
  • Assignee changed from ahmad swaid to Andrea Dell'Amico

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

You can check what plugins are installed in KNIME via

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -listInstalledRoots

and to check if the repository does work you can use

./knime -nosplash -application org.eclipse.equinox.p2.director -repository https://dl.bintray.com/silebat/fsklab_icpmf -list

Hopefully this will help to get the FSK plugin running again.

HTH,
Lars

#10 Updated by Andrea Dell'Amico 4 months ago

  • % Done changed from 20 to 80
  • Status changed from Feedback to In Progress

Lars Valentin wrote:

Hey Andrea,

could you please point me to the parts where you check if any KNIME process is running, before removing plugins? I did not see it in the ANSIBLE script nor in the CHRON job. We think it is needed that no job (KIME workflow) is running while the nodes used in this job are removed and installed again.

This is the code that you're looking for.

if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
    DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
fi

# 
if [ -d /home/gcube/tomcat/tmp/dmlocks ] ; then
   while [ ! -z "$DATAMINER_RUNNING" ] ; do
       DATAMINER_RUNNING=$( find /home/gcube/tomcat/tmp/dmlocks/ -type f )
   done
fi

The solution to repair KNIME after the incident that KNIME can't find the nodes in the repository anymore is to backup and remove the content of the p2 cache folder, in order to force KNIME to recheck all plugins repositories after KNIME execution. The folder is located under /knimefolder.../p2/org.eclipse.equinox.p2.repository/cache/

The cache removal did the trick. I'm adding a command to remove the cache after a installation failure, then the installation is going to be retried once.

#11 Updated by Andrea Dell'Amico 4 months ago

The installation of the plugin is ongoing.

#12 Updated by Andrea Dell'Amico 4 months ago

  • % Done changed from 80 to 100
  • Status changed from In Progress to Closed

The plugin is installed everywhere.

#13 Updated by Lars Valentin 4 months ago

Andrea Dell'Amico wrote:

The plugin is installed everywhere.

Great! Thanks a lot!

Also available in: Atom PDF