Better import

jkuhl commented

2026-04-08 18:10:59 +02:00

Owner

Significant time during the import is wasted with compressing and decompressing the final json file. We should determine beforehand, which files will be affected by the import of each measurement, such that we can use temporary files during the import process and only write the full files once all measurements that affect them.

implement a function that determines all files affected by import of a measurement
read out the affected files in order
at the end of the import of a measurement, write only the files that will not be affected any more
if file/dict will still be affected, hold in memory
write affected files into extra table
implement function to look up affected files in the table
write re-import for measurements in affected files in case of update

In which the last three would solve #30

Significant time during the import is wasted with compressing and decompressing the final json file. We should determine beforehand, which files will be affected by the import of each measurement, such that we can use temporary files during the import process and only write the full files once all measurements that affect them. - [x] implement a function that determines all files affected by import of a measurement - [x] read out the affected files in order - [ ] at the end of the import of a measurement, write only the files that will not be affected any more - [ ] if file/dict will still be affected, hold in memory - [ ] write affected files into extra table - [ ] implement function to look up affected files in the table - [ ] write re-import for measurements in affected files in case of update In which the last three would solve #30

jkuhl commented

2026-04-17 18:39:05 +02:00

Author

Owner

Either we keep the data as a dict until we write, otherwise we use
https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryFile

Either we keep the data as a dict until we write, otherwise we use https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryFile

jkuhl referenced this issue

2026-04-22 17:22:11 +02:00

Updates in DB #30

jkuhl commented

2026-04-22 17:34:35 +02:00

Author

Owner

The link between the parameters from the toml file and the records should be made through SQLs FOREIGN_KEYs

Rows
Columns

Better import #25