FOGL-2124 enable Foreign key constraint in SQLite #339

praveen-garg · 2021-03-25T19:25:57Z

as expected fledge_nightly_build_ub1804/614/testReport/junit/api.test_audit/ test_create_audit_entry with nonexistent (reference) log code fails now.

Bad entry tests updated in https://github.com/fledge-iot/fledge/pull/322/files

ashish-jabble

LGTM!

ashish-jabble · 2021-03-26T09:48:45Z

Also I have noticed that whenever we attempt an invalid logcode in POST audit request. Only Sqlite engine query retries upto 40 times. Is this valid?

See the below syslogs:

Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :6: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :7: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :8: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :9: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :10: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :11: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :12: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:16 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :13: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :14: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :15: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :16: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :17: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :18: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :19: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :20: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :21: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :22: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:17 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :23: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :24: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :25: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :26: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :27: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :28: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :29: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :30: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :31: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :32: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:18 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :33: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :34: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :35: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :36: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :37: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :38: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :39: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: WARNING: Connection::SQLexec - retry :40: dbHandle :C983C368: cmd :BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;: error :FOREIGN KEY constraint failed:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: ERROR: Database error after maximum retries - dbHandle :C983C368:
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: ERROR: SQLite3 storage plugin raising error: FOREIGN KEY constraint failed
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: ERROR: SQL statement: BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;
Mar 26 14:57:19 aj-ThinkPad-E450 Fledge Storage[19644]: ERROR: SQL statement: BEGIN TRANSACTION;INSERT INTO fledge.log (code, level, log) VALUES ('LOGGN_X', 2, '{"message":"Engine oil pressure low"}');COMMIT TRANSACTION;

MarkRiddoch

No we have any indication of a performance impact?

praveen-garg · 2021-03-26T10:43:08Z

I think SQLExec should handle error like SQLITE_CONSTRAINT, log and exit the loop instead of assuming BUSY/LOCKED & retrying with hard coded MAX_RETRIES 40 if rc != SQLITE_OK.

ashish-jabble · 2021-03-26T11:10:20Z

No we have any indication of a performance impact?

@MarkRiddoch Check below if makes any sense

$ cat curl-format.txt 
     time_namelookup:  %{time_namelookup}s\n
        time_connect:  %{time_connect}s\n
     time_appconnect:  %{time_appconnect}s\n
    time_pretransfer:  %{time_pretransfer}s\n
       time_redirect:  %{time_redirect}s\n
  time_starttransfer:  %{time_starttransfer}s\n
                     ----------\n
          time_total:  %{time_total}s\n

Result

ub1804

-ve scenario

$ for ((i=1;i<=5;i++)); do curl -w "@curl-format.txt" -o /dev/null -sX POST http://192.168.1.5:8081/fledge/audit -d '{"source": "LOGGN_X", "severity": "warning", "details": {"message": "Engine oil pressure low"}}'; done
     time_namelookup:  0.000024s
        time_connect:  0.000111s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000142s
       time_redirect:  0.000000s
  time_starttransfer:  4.021880s
                     ----------
          time_total:  4.022085s
     time_namelookup:  0.000032s
        time_connect:  0.000168s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000236s
       time_redirect:  0.000000s
  time_starttransfer:  4.020375s
                     ----------
          time_total:  4.020610s
     time_namelookup:  0.000035s
        time_connect:  0.000139s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000183s
       time_redirect:  0.000000s
  time_starttransfer:  4.023456s
                     ----------
          time_total:  4.023570s
     time_namelookup:  0.000036s
        time_connect:  0.000135s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000187s
       time_redirect:  0.000000s
  time_starttransfer:  4.024177s
                     ----------
          time_total:  4.024345s
     time_namelookup:  0.000036s
        time_connect:  0.000183s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000315s
       time_redirect:  0.000000s
  time_starttransfer:  4.027976s
                     ----------
          time_total:  4.028092s

+ve scenario

$ curl -w "@curl-format.txt" -o /dev/null -sX POST http://192.168.1.5:8081/fledge/audit -d '{"source": "CONAD", "severity": "warning", "details": {"message": "New configuration"}}'
     time_namelookup:  0.000053s
        time_connect:  0.000175s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000229s
       time_redirect:  0.000000s
  time_starttransfer:  0.022672s
                     ----------
          time_total:  0.022736s

Rpi with buster

-ve scenario

$ for ((i=1;i<=5;i++)); do curl -w "@curl-format.txt" -o /dev/null -sX POST http://192.168.1.6:8081/fledge/audit -d '{"source": "LOGGN_X", "severity": "warning", "details": {"message": "Engine oil pressure low"}}'; done
     time_namelookup:  0.000024s
        time_connect:  0.004010s
     time_appconnect:  0.000000s
    time_pretransfer:  0.004055s
       time_redirect:  0.000000s
  time_starttransfer:  4.063422s
                     ----------
          time_total:  4.063500s
     time_namelookup:  0.000024s
        time_connect:  0.001788s
     time_appconnect:  0.000000s
    time_pretransfer:  0.001836s
       time_redirect:  0.000000s
  time_starttransfer:  4.086205s
                     ----------
          time_total:  4.086331s
     time_namelookup:  0.000024s
        time_connect:  0.002213s
     time_appconnect:  0.000000s
    time_pretransfer:  0.002261s
       time_redirect:  0.000000s
  time_starttransfer:  4.059731s
                     ----------
          time_total:  4.059840s
     time_namelookup:  0.000027s
        time_connect:  0.002128s
     time_appconnect:  0.000000s
    time_pretransfer:  0.002176s
       time_redirect:  0.000000s
  time_starttransfer:  4.058066s
                     ----------
          time_total:  4.058169s
     time_namelookup:  0.000025s
        time_connect:  0.002095s
     time_appconnect:  0.000000s
    time_pretransfer:  0.002141s
       time_redirect:  0.000000s
  time_starttransfer:  4.055568s
                     ----------
          time_total:  4.055682s

+ve scenario

$ curl -w "@curl-format.txt" -o /dev/null -sX POST http://192.168.1.6:8081/fledge/audit -d '{"source": "CONAD", "severity": "warning", "details": {"message": "New configuration"}}'
     time_namelookup:  0.000035s
        time_connect:  0.002273s
     time_appconnect:  0.000000s
    time_pretransfer:  0.002326s
       time_redirect:  0.000000s
  time_starttransfer:  0.027674s
                     ----------
          time_total:  0.028446s

ashish-jabble · 2021-03-26T13:16:58Z

JIRA https://scaledb.atlassian.net/browse/FOGL-5197 raised so that we do not forgot to fix this issue. (Suppress warnings)

MarkRiddoch

We can not merge this with the retry issued raised by @ashish-jabble - please hold until this is resolved.

…r sqlite

enable Foreign key constraint in SQLite

6ded966

praveen-garg requested review from MarkRiddoch, sstefano69, pintomax and ashish-jabble March 25, 2021 19:26

praveen-garg mentioned this pull request Mar 26, 2021

FOGL-3125, FOGL-5121: Update audit entries tests #322

Merged

ashish-jabble previously approved these changes Mar 26, 2021

View reviewed changes

Merge branch 'develop' into FOGL-2124

57b84f2

MarkRiddoch reviewed Mar 26, 2021

View reviewed changes

Merge branch 'develop' into FOGL-2124

e2462a6

MarkRiddoch approved these changes Mar 26, 2021

View reviewed changes

MarkRiddoch requested changes Mar 26, 2021

View reviewed changes

praveen-garg mentioned this pull request Apr 15, 2021

skip test_create_nonexistent_log_code_audit_entry for sqlite until FO… #362

Merged

praveen-garg added 2 commits April 15, 2021 14:19

Merge remote-tracking branch 'origin/develop' into FOGL-2124

16c126e

removed skip tag from test_create_nonexistent_log_code_audit_entry fo…

bd3126e

…r sqlite

praveen-garg dismissed ashish-jabble’s stale review via bd3126e April 15, 2021 08:50

praveen-garg added 6 commits May 20, 2021 19:34

Merge branch 'develop' into FOGL-2124

1bc6c86

sqlite classic plugin pragma

b6ca642

Merge branch 'develop' into FOGL-2124

eb75649

Merge branch 'develop' into FOGL-2124

2ad2cf5

Merge branch 'develop' into FOGL-2124

8cc7f63

Merge branch 'develop' into FOGL-2124

fdf4e76

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FOGL-2124 enable Foreign key constraint in SQLite #339

FOGL-2124 enable Foreign key constraint in SQLite #339

praveen-garg commented Mar 25, 2021 •

edited

Loading

ashish-jabble left a comment

ashish-jabble commented Mar 26, 2021

MarkRiddoch left a comment

praveen-garg commented Mar 26, 2021

ashish-jabble commented Mar 26, 2021 •

edited

Loading

ashish-jabble commented Mar 26, 2021

MarkRiddoch left a comment

FOGL-2124 enable Foreign key constraint in SQLite #339

Are you sure you want to change the base?

FOGL-2124 enable Foreign key constraint in SQLite #339

Conversation

praveen-garg commented Mar 25, 2021 • edited Loading

ashish-jabble left a comment

Choose a reason for hiding this comment

ashish-jabble commented Mar 26, 2021

MarkRiddoch left a comment

Choose a reason for hiding this comment

praveen-garg commented Mar 26, 2021

ashish-jabble commented Mar 26, 2021 • edited Loading

ashish-jabble commented Mar 26, 2021

MarkRiddoch left a comment

Choose a reason for hiding this comment

praveen-garg commented Mar 25, 2021 •

edited

Loading

ashish-jabble commented Mar 26, 2021 •

edited

Loading