open working database in another notebook

[osagi2]

Hi,

I have the following general question/problem. I use two jupyter notebooks, one for measuring and another one for analyzing data.
Is it safe/possible to open the database in the second one while it is being updated by the first one? Is there any workaround in qcodes?

[astafan8]

Hi @osagi2 ! it is safe to have more than one process read from the same database file, but not write. This is thanks to the fact that we use sqlite database with the Write-Ahead Log (WAL) mode turned on by default. Once you'd like to write from more than one process to the same database file, then you need to be careful because that's not supported.

to make the answer closer to your use case - if the analysis notebook only reads and does not write to the database file, then everything will just work. In case in your analysis notebook, you'd like to write results to the same database file, then you can do that but only when the first notebook is not writing to the database file. If it happens that more than one process is writing to the same database file, you'll start getting errors, for example, "database locked".

In addition to this advise, you could consider using the "in memory dataset" together with the "automatic export" feature. With the in-memory dataset, the "data/numbers" of your measurement won't be continuously written to the sqlite database, those will be kept in memory, thus making the database more available for writing, and the only thing that is written to the database is "metadata" (experient name, sample name, info about parameters, the snapshot) which happens relatively quickly and only at the beginning of the measurement. And with the automatic export feature ON, at the end of the measurement the acquired data will be exported to a file on disk, usually netcdf, that means that the data will not be written into the sqlite database file, so, again, leaving the database file more available for writing. Lastly, if your analysis notebook also wishes to write data, then it can also use the "in memory dataset". Together this system has all the potential to be stable and reliable (except for the rare occasions when both the measurement and analysis notebooks intend to create datasets (thus storing "metadata" in the database file) at the very same time). For more info on the inmemry dataset and the automatic export, refer to the docs here https://qcodes.github.io/Qcodes/examples/DataSet/InMemoryDataSet.html and here https://qcodes.github.io/Qcodes/examples/DataSet/Exporting-data-to-other-file-formats.html#Export-data-automatically .

Let know if this helps and if you have more questions! :)