Testing with mongo 4.4.4 is working for now

[SebC99]

As this is not maintained anymore I was wondering if it could work with MongoDB 4.x version...
So I am testing it on a dev environment on MongoDB Atlas, with version 4.4.4 and for now, everything seems to be working fine.

Just to let you know

[SebC99]

(I'm using a patched version to make it work on 3.6 and with del commands replaced by pop as in #725)

[mahmoudajawad]

@SebC99, any update on this? Are you now using it in prod? I'm looking at the codebase and seeing that just little of updates here and there (nothing at core) is only required to get this project back on track. I'm not sure why it is abandoned when it seems to be easily maintainable.

[SebC99]

Yes and it's working perfectly for now
@RiffynInc seems to have a lot of new commits on their fork, it would be awesome if they could explain their additions.
And @tmpaul06 has developed a doc-manager for ElasticSearch 7, which is awesome too, even if I haven't tested any of these forks

[tmpaul06]

@SebC99 I have not received any feedback from the authors since I posted my comment: yougov/elastic2-doc-manager#60 (comment)

The test cases should pretty much cover the basic streaming stuff, and you can try it actively with my repo: https://github.com/tmpaul06/es7-mongo-example

I originally developed this for an internal project at my previous company. I'm not sure if they are using it in production right now.

[angeloruggieridj]

Hi, I'm trying to execute:
mongo-connector -m localhost:27017 -t http://localhost:7474/neo4j -d neo4j_doc_manager but it send me an exception (ImportError: cannot import name 'GraphError' from 'py2neo' ).
What can I do?

[tmpaul06]

ImportError: cannot import name 'GraphError' from 'py2neo'
@angeloruggieridj

That looks like a version issue. py2neo:4.x has GraphError: https://github.com/technige/py2neo/blob/py2neo-4.3.0/py2neo/database.py

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

[angeloruggieridj]

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

[tmpaul06]

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.

See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36

If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

[angeloruggieridj]

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.

See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36

If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

py2neo is 2021.0.1
neo4j-doc-manager is 0.1.1

If I try to upgrade neo4j-doc-manager to 0.1.2 I receive this error:

ERROR: Could not find a version that satisfies the requirement py2neo==2.0.8 (from neo4j-doc-manager) (from versions: 4.0.0, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.3.0, 2020.0.0, 2020.1.0, 2020.1.1, 2021.0.0, 2021.0.1)
ERROR: No matching distribution found for py2neo==2.0.8

[tmpaul06]

However that is no longer the case (it was removed) with the latest release which is v2021.0.1. I suggest that you install the specific version of py2neo using pip install py2neo==4.3.0 and try again

I've tried and the error has changed: cannot import name 'BindError' from 'py2neo' with version 4.3.0.
Upgrading at the latest release v2021.0.1, the error returns 'cannot import name 'GraphError' from 'py2neo' .

Hmm, what version of neo4j_doc_manager are you using ? From the latest one I see py2neo==2.0.8. Can you try pip list | grep py2neo and see what the output is ? Try with pip install py2neo==2.0.8 if your doc_manager is version: 0.1.2.
See https://github.com/technige/py2neo/blob/py2neo-2.0.8/py2neo/__init__.py#L36
If none of these things fix your problems, try opening a new issue with neo4j_doc_manager, I don't think your issue is related to mongo-connector at all

py2neo is 2021.0.1
neo4j-doc-manager is 0.1.1

If I try to upgrade neo4j-doc-manager to 0.1.2 I receive this error:

ERROR: Could not find a version that satisfies the requirement py2neo==2.0.8 (from neo4j-doc-manager) (from versions: 4.0.0, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.3.0, 2020.0.0, 2020.1.0, 2020.1.1, 2021.0.0, 2021.0.1)
ERROR: No matching distribution found for py2neo==2.0.8

Ahh I see the problem. The author has removed that version from pypi. You have 2 choices:

a) Install from source: See https://stackoverflow.com/questions/23521345/python-pip-how-do-i-install-a-specific-version-of-a-git-repository-from-githu , so something like pip install git+https://github.com/technige/[email protected]

b) Fork neo4j_doc_manager, upgrade it to match py2neo latest API. Frankly this is an uphill battle.

Logs from my local machine:

tharun@Tharun-XPS-9300:~$ pip install git+https://github.com/technige/[email protected]
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/technige/[email protected]
  Cloning https://github.com/technige/py2neo.git (to revision py2neo-2.0.8) to /tmp/pip-req-build-w9vfhtbr
Building wheels for collected packages: py2neo
  Building wheel for py2neo (setup.py) ... done
  Created wheel for py2neo: filename=py2neo-2.0.8-py3-none-any.whl size=154206 sha256=3f8afbb985991e65ae345dfdd47111e4d6e037d4f8ec5f916d620f8f3f091698
  Stored in directory: /tmp/pip-ephem-wheel-cache-5e9x0q2d/wheels/80/b0/24/0ebaad24ce726156942db923b0d371ae590882c818590f8aa1
Successfully built py2neo
Installing collected packages: py2neo
Successfully installed py2neo-2.0.8
WARNING: You are using pip version 20.2.2; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
tharun@Tharun-XPS-9300:~$ pip list | grep py2neo
py2neo                   2.0.8
WARNING: You are using pip version 20.2.2; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.

[angeloruggieridj]

So, I've followed your advice:

• Downgraded to py2neo 2.0.8
• Upgraded to neo4j-doc-manager to 0.1.2

So, after a restart of MongoDB and Neo4j, I've tried again to execute:
mongo-connector -v -m localhost:27017 -t http://localhost:7474/data/db -d neo4j_doc_manager
Finally, in the Terminal app, it shows Logging to /Users/angeloruggieridj/mongo-connector.log. and after 5 seconds it stops and show again the (base) user.

I've attached the file mongo-connector.log. What's the cause now?

My Neo4j DB version is 4.1.0, so not the latest version.

[tmpaul06]

So, I've followed your advice:

• Downgraded to py2neo 2.0.8
• Upgraded to neo4j-doc-manager to 0.1.2

So, after a restart of MongoDB and Neo4j, I've tried again to execute:
mongo-connector -v -m localhost:27017 -t http://localhost:7474/data/db -d neo4j_doc_manager
Finally, in the Terminal app, it shows Logging to /Users/angeloruggieridj/mongo-connector.log. and after 5 seconds it stops and show again the (base) user.

I've attached the file mongo-connector.log. What's the cause now?
mongo-connector.log

My Neo4j DB version is 4.1.0, so not the latest version.

The log clearly says that Neo4j is not reachable/running/auth is incorrect. I think if you have any further issues, you should post your questions to stackoverflow. I don't think discussion on this issue is the right approach as it will spam the issue author. One more thing: Be wary of posting your files on the public internet: it contains the name of your mongo collection as well as the username password combo for your local neo4j setup.

2021-05-03 18:25:02,034 [DEBUG] httpstream:316 - > Host: localhost:7474
2021-05-03 18:25:02,652 [INFO] httpstream:511 - < 404 Not Found [0]

[mahmoudajawad]

@tmpaul06, are you still maintaining your fork? If not, what it takes to maintain it? This seems like a lightweight project that can be maintained by anyone interested in the tech, like me, but I'm not sure the method used to maintain it. For instance, why did you need to fork it, and what have you done after forking it and continued to?

[tmpaul06]

@tmpaul06, are you still maintaining your fork? If not, what it takes to maintain it?

My goal was to merge it with upstream repo, never to keep it as a standalone project. The authors don't seem interested in the fork anyway, and their suggested method to support ES 7.x is pretty crude (doc_type name remap). Without help from the authors of the project, I cannot commit to long term maintenance.

For instance, why did you need to fork it, and what have you done after forking it and continued to?

See https://github.com/tmpaul06/elastic7-doc-manager#changelog
My previous company used this fork to move documents from mongo to ES 7.x (the latest at that time). Since I am no longer with them, I cannot confirm if they are still using it in production. As for the actual code changes, I haven't changed the core mechanism: yougov/elastic2-doc-manager@master...tmpaul06:master

If you are interested in maintaining it, feel free to do so. That is the reason why I have not published my fork to pypi. As for maintenance, you would need to match versions with elastic2-doc-manager, fix bugs as they are raised, and finally and most importantly support future versions of ES

[singhmahendrapal]

Hello Guys, Hope you are doing well!!
Today I lost my mongo-connector oplog.ts file. Is there any way to recover that one from mongodb cluster?
It's very urgent and critical for me. Please help on it.

Thanks

[mahmoudajawad]

@tmpaul06, thanks for your reply. I shall try and evaluate it. Most probably I can maintain it as long as I can use it myself.

[warby613]

While getting mongo-connector running with MongoDB 4.4, I had to use pymongo version ~-= 3.12.
Pymongo 4 introduced changes that break mongo-connector.
In case this helps anyone!