Hdd periodically write

[Daneltones8]

There is some way to reduce or remove the HDD write. Each minute application write fixed amount of data.
I pasted here the screen of disk write and my configurations that show which every minute it should does nothing.
If someone can help... Thanks

[ingoratsdorf]

Agree, same here. There's some pretty crazy disk write going on.

Running Version: 08:41:33 - Dev

1. Looks like every few minutes, it creates backups of configs in the docker mapped config folder. It's a ~150KB file that gets written every few minutes. That's the CSV backup schedule, so I switched that to 'on new device'.
2. Switched db maintenance from every 30 min to 4am once a day.

Both above did not make any difference whatsoever.

3. Increased SQLite3 cache: PRAGMA cache_size = 9000;

Did not make a difference

4. Made sure temp tables are in memory and not disk: PRAGMA temp_store = MEMORY;

Did not make any difference

5. Set logging to minimal

Did not make any difference

[jokob-sk]

Thanks for the detailed investigation and testing.

I assume this is caused by the API update method. Not sure how to optimize this though:

NetAlertX/server/api.py

Line 18 in a0db36c

def update_api(db, all_plugins, isNotification = False, updateOnlyDataSources = []):

[jokob-sk]

hi @ingoratsdorf and @Daneltones8 ,

I just pushed a new dev image (ready in ~15min) to try to decrease io ops on the HD by buffering logs a bit. Not sure if that will make a difference.

Also, can you please let me know how you are observing the Disk IO on a container? I'd like to set it up on my end so I can try to debug this further. Thanks in advance!

[ingoratsdorf]

I use proxmox with InfluxDB. Proxmox writes stats about system and containers to InfluxDB every 10sec.

[Daneltones8]

Thank you! I will try it as soon as I will have a bit of time. About displaying disk IO, I use QNAP nas which has this functionalities integrated on container station. Il ven 13 dic 2024, 22:49 jokob-sk ***@***.***> ha scritto:

…

hi @ingoratsdorf <https://github.com/ingoratsdorf> and @Daneltones8 <https://github.com/Daneltones8> , I just pushed a new dev image (ready in ~15min) to try to decrease io ops on the HD by buffering logs a bit. Not sure if that will make a difference. Also, can you please let me know how you are observing the Disk IO on a container? I'd like to set it up on my end so I can try to debug this further. Thanks in advance! — Reply to this email directly, view it on GitHub <#684 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BF3GIUJJJK7TDZAZDRYRL3T2FNI5LAVCNFSM6AAAAABTQQSAGGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJWGIZDIMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ingoratsdorf]

As per #914 (comment), the latest dev from today seem to have fixed the issue to the largest part. Still lots of writing considering there is a cached SQLite3 working in the background. Dunno if it's that or something else. Are you writing any temp files?

Homeassistant writes 100x more data to MariaDB but that has a waaay lower disk write access, about 1/30.

[jokob-sk]

Truth be told - HomeAssistant has a bigger dev community vs only one dev for the core backend in NAX(me lol) and NAX is build on an not so scalable architecture from the upstream original project. In order to reduce DB locks, which were already occurring on the original project, in NAX I'm exposing a JSON API layer, however these file scan grow quite quickly. These regular writes are I think when these files are updated.

With the next version I'm introducing a GraphQL server/layer for Settings and Devices. Other endpoints will hopefully migrated in future. Next could be accessing the DB directly via the GraphQL layer as it still relies on those json endpoints.

I also think the SqliteDB isn't great for parallel R/W operations.

I am trying to balance features vs scalability7 vs stability on the project, and my personal life lol

Happy of course for anyone more skilled to help reduce these writes on improve on the architecture.

[ingoratsdorf]

No blame, no worries. :-)
I know that game.

Not sure if this would be an option, but if the API JSON files do not need to be persistent, could they exist only in ram?

As in
mount -t tmpfs tmpfs /app/api

https://www.man7.org/linux/man-pages/man5/tmpfs.5.html