Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error at restart with a high number of services #1315

Open
geofjamg opened this issue Mar 20, 2024 · 5 comments
Open

Error at restart with a high number of services #1315

geofjamg opened this issue Mar 20, 2024 · 5 comments

Comments

@geofjamg
Copy link
Contributor

geofjamg commented Mar 20, 2024 •
edited
Loading

I have 150 south services and 10 north services and when I stop then start fledge I have the following error:

Mar 20 10:10:48 machine1 Fledge [1990997] INFO: script.fledge: Fledge started.
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.common.configuration_manager: Unable to create new category based on category_name iec104_s99 and category_description iec104_s99 an
d category_json_schema {'plugin': {'description': 'iec104 south plugin', 'type': 'string', 'readonly': 'true', 'default': 'iec104', 'value': 'iec104'}, 'asset': {'description': 'Asset name', 'type':
 'string', 'order': '1', 'displayName': 'Asset Name', 'mandatory': 'true', 'default': 'iec104', 'value': 'iec104'}, 'protocol_stack': {'description': 'protocol stack parameters', 'type': 'JSON', 'or
der': '2', 'displayName': 'Protocol stack parameters', 'default': '{ "protocol_stack" : { "name" : "iec104client", "version" : "1.0", "transport_layer" : { "redundancy_groups" : [ { "connections" : 
[ { "srv_ip" : "127.0.0.1", "port" : 2404 } ], "rg_name" : "red-group1", "tls" : false, "k_value" : 12, "w_value" : 8, "t0_timeout" : 10, "t1_timeout" : 15, "t2_timeout" : 10, "t3_timeout" : 20 } ] 
}, "application_layer" : { "orig_addr" : 10, "ca_asdu_size" : 2, "ioaddr_size" : 3, "asdu_size" : 0, "gi_time" : 60, "gi_cycle" : 60, "gi_all_ca" : false, "utc_time" : false, "cmd_wttag" : false, "c
md_parallel" : 0, "time_sync" : 0 } } }', 'value': '{ "protocol_stack" : { "name" : "iec104client", "version" : "1.0", "transport_layer" : { "redundancy_groups" : [ { "connections" : [ { "srv_ip" : 
"127.0.0.1", "port" : 2404 } ], "rg_name" : "red-group1", "tls" : false, "k_value" : 12, "w_value" : 8, "t0_timeout" : 10, "t1_timeout" : 15, "t2_timeout" : 10, "t3_timeout" : 20 } ] }, "application
_layer" : { "orig_addr" : 10, "ca_asdu_size" : 2, "ioaddr_size" : 3, "asdu_size" : 0, "gi_time" : 60, "gi_cycle" : 60, "gi_all_ca" : false, "utc_time" : false, "cmd_wttag" : false, "cmd_parallel" : 
0, "time_sync" : 0 } } }'}, 'exchanged_data': {'description': 'exchanged data list', 'type': 'JSON', 'order': '3', 'displayName': 'Exchanged data list', 'default': '{ "exchanged_data": { "name" : "i
ec104client", "version" : "1.0", "datapoints" : [ { "label":"TM-1", "protocols":[ { "name":"iec104", "address":"41025-4202832", "typeid":"M_ME_NA_1" } ] }, { "label":"TM-2", "protocols":[ { "name":"
iec104", "address":"41025-4202852", "typeid":"M_ME_NA_1" } ] }, { "label":"TS-1", "protocols":[ { "name":"iec104", "address":"41025-4206948", "typeid":"M_SP_TB_1" } ] } ] } }', 'value': '{ "exchange
d_data": { "name" : "iec104client", "version" : "1.0", "datapoints" : [ { "label":"TM-1", "protocols":[ { "name":"iec104", "address":"41025-4202832", "typeid":"M_ME_NA_1" } ] }, { "label":"TM-2", "p
rotocols":[ { "name":"iec104", "address":"41025-4202852", "typeid":"M_ME_NA_1" } ] }, { "label":"TS-1", "protocols":[ { "name":"iec104", "address":"41025-4206948", "typeid":"M_SP_TB_1" } ] } ] } }'}
, 'tls': {'description': 'tls parameters', 'type': 'JSON', 'order': '4', 'displayName': 'TLS parameters', 'default': '{ "tls_conf" : { "private_key" : "iec104_client.key", "own_cert" : "iec104_clien
t.cer", "ca_certs" : [ { "cert_file": "iec104_ca.cer" }, { "cert_file": "iec104_ca2.cer" } ], "remote_certs" : [ { "cert_file": "iec104_server.cer" } ] } }', 'value': '{ "tls_conf" : { "private_key"
 : "iec104_client.key", "own_cert" : "iec104_client.cer", "ca_certs" : [ { "cert_file": "iec104_ca.cer" }, { "cert_file": "iec104_ca2.cer" } ], "remote_certs" : [ { "cert_file": "iec104_server.cer" 
} ] } }'}}
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: Failed to create category.
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: Traceback (most recent call last):
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/services/core/api/configuration.py"
, line 149, in create_category#012    await cf_mgr.create_category(category_name=category_name, category_description=category_desc,
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/configuration_manager.py", l
ine 1280, in create_category#012    all_categories = await self._read_all_category_names()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/configuration_manager.py", l
ine 616, in _read_all_category_names#012    results = await self._storage.query_tbl_with_payload('configuration', payload)
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/storage_client/storage_clien
t.py", line 308, in query_tbl_with_payload#012    jdoc = await resp.json()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", li
ne 1098, in json#012    await self.read()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", li
ne 1036, in read#012    self._body = await self.content.read()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 375
, in read#012    block = await self.readany()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 397
, in readany#012    await self._wait("readany")
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 304
, in _wait#012    await waiter
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Mar 20 10:11:31 machine1 Fledge iec104_s99[1994560]: ERROR: Failed to add configuration category: Response payload is not completed.

I also have sometimes this kind of crash:

Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: Signal 6 (Aborted) trapped:
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (0) 0       0x5561e7f6e9b2 handler(int) + 82---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (1) 1       0x7f55bff59090 /lib/x86_64-linux-gnu/libc.so.6(+0x43090) [0x7f55bff59090]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (2) 2       0x7f55bff5900b gsignal + 203---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (3) 3       0x7f55bff38859 abort + 299---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (4) 4       0x7f55c01c38d1 /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e8d1) [0x7f55c01c38d1]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (5) 5       0x7f55c01cf37c /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7f55c01cf37c]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (6) 6       0x7f55c01cf3e7 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7f55c01cf3e7]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (7) 7       0x7f55c01cf699 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa699) [0x7f55c01cf699]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (8) 8       0x7f55c068d9ec /home/gj/sources/fledge/cmake_build/C/lib/libcommon-lib.so.1(+0x539ec) [0x7f55c068d9ec]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (9) 9       0x5561e7f74890 SouthService::createConfigCategories(DefaultConfigCategory, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 208---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (10) 10      0x5561e7f74e06 SouthService::loadPlugin() + 582---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (11) 11      0x5561e7f757d6 SouthService::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned short) + 1014---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (12) 12      0x5561e7f5d8c7 main + 631---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (13) 13      0x7f55bff3a083 __libc_start_main + 243---------
Mar 19 13:54:18 machine1 Fledge[720832] ERROR: logger: fledge.services.core.api.configuration: Failed to create the child relationship for South category.

Any idea what could be the issue and if there is a way to workaround ?
@MarkRiddoch
Copy link
Contributor

I suspect this is all triggered by the services starting together and overloading the API of the core service. We have done the first state of adding the ability to order the start up, which will help with this. But we need to probably complete that work south that we can stagger the startup of the south services. Or maybe even have a mechanism that automatically rate limits the service startup when we start getting large numbers.

@geofjamg
Copy link
Contributor Author

I tried with just one south service configured and when I stop then restart fledge I get more than 300 hundreds http requests across all services just for them to restart (there is non data acquisition). I excluded ping http requests.

POST /fledge/service HTTP/1.1
GET /fledge/service/category/Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Advanced/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
..POST /fledge/interest HTTP/1.1
.PUT /storage/reading/query HTTP/1.1
PUT /storage/table/streams/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /storage/table/scheduled_processes HTTP/1.1
GET /storage/table/schedules HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/ping HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
.GET /fledge/ping HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/statistics/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Storage/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Storage/children HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
PUT /fledge/service/category/Storage/plugin HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /fledge/service/category/Storage/readingPlugin HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/asset_tracker/query HTTP/1.1
PUT /storage/table/alerts/query HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /storage/table/log HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Utilities/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
GET /fledge/service/category/purge_system HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/ping HTTP/1.1
PUT /storage/table/statistics/query HTTP/1.1
Fledge started.
GET /fledge/service/ping HTTP/1.1
GET /fledge/service/ping HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/South/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1Advanced HTTP/1.1
POST /fledge/service HTTP/1.1
POST /storage/table/log HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/interest HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
GET /fledge/service/category/iec104_pivot_filter HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1%20Filters/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1_iec104_pivot_filter HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1Security HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /storage/reading HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1

@geofjamg
Copy link
Contributor Author

geofjamg commented Apr 2, 2024

@MarkRiddoch I found the issue. Simple-Web-Server has a default request timeout of 5s:
image
This is a little bit short when restarting 160 services because the storage server is flooded of request during a few seconds.
I tried to increase it and all my services finally restart without any error.
Would it be ok for you to be able to configure in fledge the the timeout_request and the timeout_content so that we can adapt it better to use cases?

@geofjamg
Copy link
Contributor Author

geofjamg commented Apr 2, 2024

The hack I did to make it working:
image

@MarkRiddoch
Copy link
Contributor

MarkRiddoch commented Apr 3, 2024 •
edited
Loading

A new configuration item has been added to the Advanced - Storage category that allows this timeout to be set. The default has also been increased. This is in the develop branch and will be in tonights nightly build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@geofjamg @MarkRiddoch