Registry prod deploy failed

17321
Reporter: mdoering
Assignee: fmendez
Type: Bug
Summary: Registry prod deploy failed
Priority: Critical
Resolution: Fixed
Status: Closed
Created: 2015-02-26 12:18:01.11
Updated: 2015-02-26 19:33:55.438
Resolved: 2015-02-26 19:33:55.413
        
Description: The registry only deploy to production failed this morning:
http://builds.gbif.org/job/markus-deploy/13/console

The new 2.25 release was deployed and the jetty service was running fine. The varbsih backend was corretly set to the new port 9014 too, but somehow varnish returned 503 errors for dataset details for example. A simple varnsih restart solved that problem.

In addition the apache registry VH was not updated and still referred to the old port.

The deploy logs claim to have removed the new service but it did not, maybe that is related to the not updated apache and missed varnish restart?

The running instance after the deploy uses the 2.25 instance folder which should be the new release as the previous jetty was deployed with version 2.24]]>
    

Author: fmendez@gbif.org
Created: 2015-02-26 14:58:54.236
Updated: 2015-02-26 15:12:07.66
        
In that log file there are 2 errors thrown by Ansible processing the template default.vcl:
atal: [api.gbif.org] => {'msg': 'AnsibleUndefinedVariable: One or more undefined variables: the inline if-expression on line 63 evaluated to false and no else section was defined.', 'failed': True}
fatal: [api.gbif.org] => {'msg': 'AnsibleUndefinedVariable: One or more undefined variables: the inline if-expression on line 63 evaluated to false and no else section was defined.', 'failed': True}

That error was introduced in this line: https://github.com/gbif/c-deploy/blob/master/services/roles/varnish/templates/default.vcl#L63,  the ternary 'if' operator has to be changed from: {{ '/sandbox' if env != 'prod' }}  to {{ '/sandbox' if env != 'prod' else ''}}

The error reported in the logs was fixed here: https://github.com/gbif/c-deploy/commit/8374c2df43def8d8ee586a8b3dc598d3a75c963c