Handle SQL deadlock termination in stale audit & aggregate processors

XMLWordPrintable

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major
    • Component/s: DB, SolarJobs
    • None

      The stale audit/aggregate processor jobs run in parallel, and might encounter a deadlock termination event, surfaced as a org.springframework.dao.DeadlockLoserDataAccessException exception. This exception could be caught and either ignored or have the operation re-tried, instead of logging an ERROR message. An example stacktrace is

      org.springframework.dao.DeadlockLoserDataAccessException: ConnectionCallback; SQL []; ERROR: deadlock detected Detail: Process 98124 waits for ShareLock on transaction 2232825783; blocked by process 76896. Process 76896 waits for ShareLock on transaction 2232825782; blocked by process 98124. Hint: See server log for query details. Where: while deleting tuple (612,1) in relation "_hyper_12_1828_chunk" SQL statement "DELETE FROM solaragg.aud_datum_monthly WHERE node_id = stale.node_id AND source_id = stale.source_id AND ts_start > stale.ts_start - interval '1 month' AND ts_start < stale.ts_start + interval '1 month' AND ts_start <> stale.ts_start" PL/pgSQL function solaragg.process_one_aud_datum_daily_stale(character) line 125 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: deadlock detected Detail: Process 98124 waits for ShareLock on transaction 2232825783; blocked by process 76896. Process 76896 waits for ShareLock on transaction 2232825782; blocked by process 98124. Hint: See server log for query details. Where: while deleting tuple (612,1) in relation "_hyper_12_1828_chunk" SQL statement "DELETE FROM solaragg.aud_datum_monthly WHERE node_id = stale.node_id AND source_id = stale.source_id AND ts_start > stale.ts_start - interval '1 month' AND ts_start < stale.ts_start + interval '1 month' AND ts_start <> stale.ts_start" PL/pgSQL function solaragg.process_one_aud_datum_daily_stale(character) line 125 at SQL statement 
          at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:263) 
          ...
      

            Assignee:
            Matt Magoffin
            Reporter:
            Matt Magoffin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: