What behavior do we want?

• Replace the file on disk and include in db?
• Keep original file and just include in db?
• Should we add the use of a force flag?

Keep the original and include it in the database. I don't mind it being a force flag really - I believe that this situation only happens for manual stuff - so a force might be useful :)

Intuitively I would think that there should not be any files present in the housekeeper directories if they have not been added through the API.

Can we clarify the manual stuff this happens with? @karlnyr

moving description over from a duplicated issue:
Description
housekeeper add file fails, stating that the file already exists. However, when the specific bundle is retrieved with housekeeper get bundle it is shown to be empty. When looking at the bundle directory, the file is present in a version - so it should be listed for the bundle. It cannot be retrieved with housekeeper get file either.

The command below was run in the /home/proj/production/housekeeper-bundles/ADM1091A3/2018-06-05 directory:

for f in *; do housekeeper add file -t fastq -t H9GA6ADXX -b ADM1091A3 ./${f}; done
2023-06-13 09:47:51 hasta.scilifelab.se housekeeper.cli.core[37109] INFO Use root path /home/proj/production/housekeeper-bundles
2023-06-13 09:47:51 hasta.scilifelab.se housekeeper.cli.add[37109] INFO Running add file
2023-06-13 09:47:51 hasta.scilifelab.se housekeeper.store.api.handlers.read[37109] INFO Fetching bundle with name: ADM1091A3
Traceback (most recent call last):
File "/home/proj/production/bin/miniconda3/envs/P_main/bin/housekeeper", line 8, in
sys.exit(base())
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/housekeeper/cli/add.py", line 124, in file_cmd
link_to_relative_path(version=version, file_path=file_path, root_path=context.obj[ROOT])
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/housekeeper/include.py", line 63, in link_to_relative_path
link_file(file_path=file_path, new_path=housekeeper_path, hardlink=True)
File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/housekeeper/include.py", line 19, in link_file
os.link(file_path.resolve(), new_path)
FileExistsError: [Errno 17] File exists: '/home/proj/production/housekeeper-bundles/ADM1091A3/2018-06-05/ADM1091A3_L001_R1_001.fastq.gz' -> '/home/proj/production/housekeeper-bundles/ADM1091A3/2018-06-05/ADM1091A3_L001_R1_001.fastq.gz'

Though I agree that ideally we should avoid manually modifying the database, I have also found this issue and wondered if having a --force or --skip-hard-linking flag would be useful when doing manual work.
The problem arose when manually processing old flow cells stored on disk but with missing files in the housekeeper bundle or filtering vcf files from balsamic cases that fail for having too many variants as it is often more straightforward to find the necessary input files already in the housekeeper bundle. I found a workaround by moving or generating the files on a different directory and then adding them to housekeeper.

Suggested solution: Before hard linking the file, check if there is a file present. If so use Path.samefile() to compare them. If True, only add to the Database. This might be a bit cumbersome but avoids the problem of overwriting anything already in the bundle directory.

Note: With the current implementation, one can already achieve this by using the flag --keep-input-path.