For me, it took a while to understand correct MySQL commands to clean up failed runs. Anyway, I clean up all failed runs in MySQL DB and change their status from analysis error to ready to analyzer. Then autoanalyzer pick up these runs immediately. Still I need to time to figure out an automatic recovering procedure for the main analyzer and post-processing analysis (promptsummary, regression, and so on)
Run:13827 SegNumber: 4 Status: analysis error[ -12] Host:cdaq [ 16212] Run:13828 SegNumber: 3 Status: analysis error[ -12] Host:cdaq [ 16212] Run:13829 SegNumber: 9 Status: analysis error[ -12] Host:cdaq [ 16156] Run:13830 SegNumber: 3 Status: analysis error[ -12] Host:cdaq [ 15556] Run:13832 SegNumber: 3 Status: analysis error[ -12] Host:cdaq [ 16212] Run:13960 SegNumber: 8 Status: analysis error[ -12] Host:farm [129577018] Run:13985 SegNumber:12 Status: analysis error[ -12] Host:farm [129577018] Run:14079 SegNumber:13 Status: analysis error[ -12] Host:farm [129577018]
Run:13827 SegNumber: 4 Status: farm job in progress[ 50] Host:farm [129577018] Run:13828 SegNumber: 3 Status: farm job in progress[ 50] Host:farm [129577018] Run:13829 SegNumber: 9 Status: farm job in progress[ 50] Host:farm [129577018] Run:13830 SegNumber: 3 Status: analysis in progress[ 20] Host:cdaq [ 19938] Run:13832 SegNumber: 3 Status: farm job in progress[ 50] Host:farm [129577018] Run:13960 SegNumber: 8 Status: ready to analyze[ 19] Host:none [ 0] Run:13985 SegNumber:12 Status: ready to analyze[ 19] Host:none [ 0] Run:14079 SegNumber:13 Status: ready to analyze[ 19] Host:none [ 0]