Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to Prosopo Procaptcha #372

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Migrate to Prosopo Procaptcha #372

wants to merge 18 commits into from

Conversation

forgetso
Copy link

@forgetso forgetso commented Feb 22, 2024

Fixes #337

PR to migrate rococo faucet from reCAPTCHA to Prosopo Procaptcha, which runs on the rococo test network.

Overview

When using Procaptcha, the flow of data in the faucet client app is as follows :

Frontend Flow

  1. The user enters the page.
  2. The user selects their network and how many tokens they require.
  3. The user clicks the Procaptcha check box.
  4. A Polkadot user_account is generated using fingerprinting as random entropy
  5. The user_account and the faucet's site_key are passed to the Procaptcha Contract in rococo, and details of a random Captcha Provider are returned.
  6. The rococo network is contacted, querying the Procaptcha Contract, and a Captcha Provider is returned
  7. The website asks the Captcha Provider for a challenge, passing the site_key, user_account, block_number, and other data related to the Captcha Provider.
  8. The Captcha Provider checks it was randomly selected by the Procaptcha Contract, by submitting the same read performed in step 5. using historical block data (e.g. queryAt(block_number))
  9. Assuming step 8. is successful, the Captcha Provider returns a challenge
  10. The user completes the challenge, and passes the solution to the Captcha Provider. If the solution is correct, the checkbox becomes ticked, a challenge payload is added to the form, and the form submission button becomes enabled
  11. The user presses the form submission button and the form data + captcha payload are sent to the faucet backend

Additional Info

Page Reloads

We noticed that the reCAPTCHA implementation would ask the user for a new captcha on every page load. We have recreated this functionality using procaptcha by clearing any local storage on load of the CaptchaV2.svelte component. This prevents a locally stored Captcha Provider from being contacted on page reload, and passing the user_account due to their historical records stored on the Captcha Provider.

Max Verified Time

The maximum amount of time a captcha solution is valid for is 60s. This can be changed using the env variable PROCAPTCHA_MAX_VERIFIED_TIME, which takes a millisecond amount.

Backend Flow

We'll begin the backend flow from where the frontend flow left off.

  1. The backend receives the form data from the frontend.
  2. The backend takes the Procaptcha payload and passes it to a TypeScript package called @prosopo/server.
  3. This package performs the following checks
    a. Check if the provider was actually chosen at block_number
    b. If a provider_url is passed with the payload, the Captcha Provider is contacted to verify the user. Otherwise, read the user_account score from the contract directly after checking the time since the last correct captcha is less than the maxVerifiedTime (60s, mentioned previously)
  4. After 3a. or 3b. the user is either verified via a Captcha Provider or the Procaptcha Contract and will be permitted or denied their funds.

Testing

Client

Our client bundle continues to reach out to the rococo network for a random provider and the provider sends a real captcha. Solving this is clearly out of scope for UI tests so the provider is patched by injecting our mock provider (https://mockprovider.prosopo.io) into local storage. The mock provider always returns { "success": true } for the Alice account and the test site_key=5C4hrfjw9DjXZTzV3MwzrrAr9P1MJhSrvWGWqi1eSuyUpnhM, which is the all zero address.

E2E

As we are using a contract to return a provider, we implemented an additional contracts parachain in the E2E tests. This requires us to do the following:

  1. deploy the Procaptcha Contract
  2. register a Captcha Provider in the contract (We register our mock provider - https://mockprovider.prosopo.io)
  3. store a Captcha Provider dataset in the contract
  4. register an app (site_key) in the contract

Then when a random provider is selected, it is always the mock provider. Again, passing a user_account of Alice's address and the zero address for site_key returns { "success": true }. Obviously, the downside here is the time taken for transactions in 12s parachains in zombienet. We have tried to use the substrate-contracts-node, which has instant seal, alongside the zombienet configuration but didn't succeed.

We would appreciate any feedback on the approach so far!

forgetso and others added 2 commits February 22, 2024 12:36
* working frontend temp

* temp

* package json fix

* npm version bump

* Drip request type changing to procaptcha

* Drip request type changing to procaptcha

* swapping all occurrences of recaptcha with procaptcha and adding captcha logic

* Renaming recaptcha files and updating imports

* adding Prosopo outputs to tests

* bugtesting

* Removing bundle and adding back regular tests

* changing frontend to use remote bundle

* Add CAPTCHA_PROVIDER config item

* add feature flag for captcha provider and update tests for procaptcha

* Add react deps

* Add test configs for both captcha providers

* lint:fix and yarn install

* add Paseo testnet config

* remove wococo testnet config

* Adding forgotten paseo deployment

* Address PR comments

* Fix dummy body

* Add debug

* Bump the npm_and_yarn group across 1 directories with 1 update (paritytech#368)

* Create e2e test environment for procaptcha

* remove rogue file

* Update zombienet config in workflow

* yarn.lock

* yarn.lock

* downgrade jest

* try fix for buildcheck

* Pin polkadot versions

* yarn.lock with 1.22.21

* try new yarn.lock

* Update faucet lockfile

* Procaptcha dev (#2)

* dev setup wip

* update docker setup for procaptcha

* Remove extra unecessary stuff

* Undo formatting

* Update mock provider URL to use prosopo subdomain

* bump prosopo versions

* add missing dep

* Update to working package versions

* Upgrade prosopo deps

* Add missing body parameter

---------

Co-authored-by: Hugh <[email protected]>
Co-authored-by: hugh <[email protected]>
Co-authored-by: Pierre Besson <[email protected]>
Co-authored-by: Yuri Volkov <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@forgetso forgetso requested review from a team as code owners February 22, 2024 12:41
@cla-bot-2021
Copy link

cla-bot-2021 bot commented Feb 22, 2024

User @HughParry, please sign the CLA here.

Copy link
Contributor

@Bullrich Bullrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @forgetso! Thank you for the PR.

First of all, thank you a lot for adding tests! That's amazing!

More detail to follow...

Does this mean that the PR is still draft?


@mutantcornholio could you please look into the faucet (not client) code? I would appreciate a second set of eyes


- `PUBLIC_CAPTCHA_KEY`: The [reCaptcha v2 site key](https://www.google.com/u/0/recaptcha/admin).
- `PUBLIC_CAPTCHA_PROVIDER`: The captcha provider. Currently `procaptcha` and `recaptcha` are supported. You will then need one of the following site keys:
- `PUBLIC_PROSOPO_SITE_KEY`: The [Prosopo site key](https://prosopo.io/) which is `5HUBceb4Du6dvMA9BiwN5VzUrzUsX9Zp7z7nSR2cC1TCv5jg`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this key generated by you? If we want to change it we need to create an account and update it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is generated by us but you can generate your own by registering on our website. I think it would be best if we hop on a call to go through everything next week. I'll reach out to you on Element to arrange.

client/package.json Show resolved Hide resolved
client/package.json Outdated Show resolved Hide resolved
@forgetso
Copy link
Author

Hi @forgetso! Thank you for the PR.

First of all, thank you a lot for adding tests! That's amazing!

No, problem. It wouldn't be a proper integration without them although it was a mission to build them!

More detail to follow...

Does this mean that the PR is still draft?

It's 99% of the way there, I plan to write a proper overview of everything that's changed and why because there are a lot of changes. The only reason I'm not saying 100% is because there is an occasionally failing test and I can't work out why it's happening.

I'll try to get the summary done now so that you guys can review with additional context.

@forgetso
Copy link
Author

I've added a summary of the changes @Bullrich, hope it helps 🙂

Copy link
Contributor

@Bullrich Bullrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for updating the body of the PR. Everything is very detailed.

I haven't find any problems with the code in the client directory, but, I have a couple of questions:

  1. I tried to create an account in https://prosopo.io/#signup but I never received the email. Could you please look into this? I would like to be able to manage the key and store it as a secret.
image
  1. I'm unable to test the client in my computer.
    • When loading the project (yarn dev) and clicking on the captcha, I'm getting the error ProviderUrlUsed. I haven't modified any file, any idea how/what could be causing this?

e2e/zombienet.native.toml Outdated Show resolved Hide resolved
src/dripper/DripRequestHandler.ts Outdated Show resolved Hide resolved
src/dripper/polkadot/PolkadotActions.ts Outdated Show resolved Hide resolved
@Bullrich Bullrich added the client Changes related to th client label Feb 27, 2024
@forgetso
Copy link
Author

I tried to create an account in https://prosopo.io/#signup but I never received the email. Could you please look into this? I would like to be able to manage the key and store it as a secret.

Hey @Bullrich, I think the email might have been delayed. Have you now received it?

I'm unable to test the client in my computer.
When loading the project (yarn dev) and clicking on the captcha, I'm getting the error ProviderUrlUsed. I haven't modified any file, any idea how/what could be causing this?

The contract that's on rococo is slightly out of date with the one in our sources. We plan to update this very soon. The error you should have received is DappDoesNotExist because your local site key is not registered in our live contract on rococo. You can try using the site key that you should hopefully now have received by email instead. Please let me know if this works!

Copy link
Contributor

@mutantcornholio mutantcornholio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a whole bunch of questions, but there is an independent big one: I don't think we need two different captcha systems. Let's remove recaptcha completely. If Procapthca won't work for some reason, we can revert the changes, but I can't see a reason in supporting two.

package.json Outdated Show resolved Hide resolved
package.json Show resolved Hide resolved
env.faucet.config.json Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
src/dripper/Procaptcha.ts Outdated Show resolved Hide resolved
src/faucet.e2e.ts Outdated Show resolved Hide resolved
src/test/setupE2E.ts Outdated Show resolved Hide resolved
src/test/setupE2EProcaptcha.ts Outdated Show resolved Hide resolved
src/test/setupE2EProcaptcha.ts Outdated Show resolved Hide resolved
src/dripper/Procaptcha.spec.ts Outdated Show resolved Hide resolved
@forgetso
Copy link
Author

forgetso commented Mar 1, 2024

I don't think we need two different captcha systems

Hey @mutantcornholio, I'd be happy to remove the ad tracking reCAPTCHA out of the faucet app. The option to have 2 captcha systems was implemented due to previous request in the issue. Maybe you guys could discuss and let me know which way you want to go.

Cheers

@mutantcornholio
Copy link
Contributor

I see. I actually missed that. We had an internal discussion now and our opinions have diverged (and looks like I'm in minority).
Soo, let's keep recaptcha for now.

forgetso and others added 2 commits March 26, 2024 22:57
* lockfile update

* Upgrade deps

* Package-lock.json

* Updating e2e tests readme (paritytech#375)

* Fixing e2e readme link

* Adding runtimeRestarter (paritytech#376)

Fixes paritytech#223

+ Adding prom-client dependency, because it was removed from
@eng-automation/js (rightfully so)

* Updating zombinenet verion in e2e docs (paritytech#378)

@josepot has reported an error with `@zombienet/[email protected]`, while the
latest version worked

* Using PAPI for e2e tests (paritytech#379)

* Currently, e2e types are generated on postinstall and prebuild steps,
which requires metadata to be available in Docker image, even though
it's not technically used in the application code. I think separating
things would make things more complicated, but it's possible.
* Couldn't undestand the reason for having two different COPY
instructions
in Dockerfile, but that doesn't live well with generating types in
postinstall, so merging those two
* .scale metadata files from Zombienet hosts saved in the codebase,
in order to be able to compile the code separately from running
Zombienet

* Update dependencies

* yarn.lock

* Bump the npm_and_yarn group across 2 directories with 2 updates (paritytech#381)

* removed base url (paritytech#382)

This removes the base url, making the website work on the root of the
domain.

Also, by doing this change, and having set up the dns, this resolves
paritytech#348

* Update deps

* Update API calls

* Silence polkadot API type errors

* Update captcha type to image and set width of captcha container

* remove grid class

* switch back to PJS. change port number

* remove prettier deps

* update port

* set procaptcha details

* fix dropdown z-index

* Fix z-indexes

* Change mock provider version

* remove console.errors

---------

Co-authored-by: Yuri Volkov <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Javier Bullrich <[email protected]>
@Bullrich
Copy link
Contributor

Hi @forgetso!

How's the PR coming along?

Has all the requested changes been addressed?

I know it's troublesome, but could you also resolves the conflicts please? Once this is done we'll jump to give it another review 🙂

Thanks!

@forgetso
Copy link
Author

Hey @Bullrich, apologies for the radio silence. I actually have a second PR in progress which will merge into our forked copy. This addresses all of the issues and merges all upstream changes. The most recent upstream changes have caused a bit of difficulty due to an upstream polkadot-sdk bug. However, I'm hoping to have an update to the PR for you to review this week.

forgetso and others added 7 commits April 29, 2024 12:53
* Updating e2e tests readme (paritytech#375)

* Fixing e2e readme link

* Adding runtimeRestarter (paritytech#376)

Fixes paritytech#223

+ Adding prom-client dependency, because it was removed from
@eng-automation/js (rightfully so)

* Updating zombinenet verion in e2e docs (paritytech#378)

@josepot has reported an error with `@zombienet/[email protected]`, while the
latest version worked

* Using PAPI for e2e tests (paritytech#379)

* Currently, e2e types are generated on postinstall and prebuild steps,
which requires metadata to be available in Docker image, even though
it's not technically used in the application code. I think separating
things would make things more complicated, but it's possible.
* Couldn't undestand the reason for having two different COPY
instructions
in Dockerfile, but that doesn't live well with generating types in
postinstall, so merging those two
* .scale metadata files from Zombienet hosts saved in the codebase,
in order to be able to compile the code separately from running
Zombienet

* Bump the npm_and_yarn group across 2 directories with 2 updates (paritytech#381)

* removed base url (paritytech#382)

This removes the base url, making the website work on the root of the
domain.

Also, by doing this change, and having set up the dns, this resolves
paritytech#348

* Added markup code generation (paritytech#383)

It generates a JSON object inside the header following the instructions
from:
https://developers.google.com/search/docs/appearance/structured-data/faqpage

It parses the information of the markdown text to generate this json
object.

This resolves paritytech#277

* Updated actions to latest and added retention period (paritytech#384)

We currently don't have an artifact retention period, so it defaults to
90 days.

While the artifact for the site is not too big, having one build per
push and PR can quickly accumulate the amount and make us hit our hard
limit.

By adding a limit to the retention day of 5 days we ensure that we get
rid of the old artifacts faster than waiting for two whole months.

I also updated all the actions version to latest release

* Bump the npm_and_yarn group across 1 directory with 1 update (paritytech#385)

* Bump the npm_and_yarn group across 1 directory with 1 update (paritytech#387)

* Update Frequency Faucet URL (paritytech#393)

As the Frequency Faucet now also supports Frequency Paseo testnet in
addition to the Rococo one, the url is being set to a generic one
instead of specific for the network.

https://faucet.testnet.frequency.xyz/

* moved tag-client to work in pull_request_target (paritytech#394)

It fails when an external contributors make a PR.

This will fix that issue.

* Merge latest upstream and use a transaction queue to send txs to nodes

* Add MIT LICENSE (paritytech#395)

* bump to polkadot 10.13.1

* lockfile

* yarn.lock

* yarn.lock

* yarn.lock

* yarn.lock

* yarn.lock client

* Fix test mock

* Add @polkadot/util to dev dependencies

* Address comments from @mutantcornholio

---------

Co-authored-by: Yuri Volkov <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Javier Bullrich <[email protected]>
Co-authored-by: Wil Wade <[email protected]>
Co-authored-by: Maksym H <[email protected]>
* link to docs

* Add missing setter
@forgetso
Copy link
Author

Hey @Bullrich, I think the PR should now pass if you run the tests. I'll keep checking for updates. Cheers

@mutantcornholio
Copy link
Contributor

@forgetso the unit tests run has failed with

FAIL src/dripper/Procaptcha.spec.ts (12.627 s)
  ● Console
    console.error
      ERROR (app): AxiosError {
        code: 'ERR_BAD_RESPONSE',
        config: {
          adapter: [ 'xhr', 'http' ],
          data: '{"dapp":"5HUBceb4Du6dvMA9BiwN5VzUrzUsX9Zp7z7nSR2cC1TCv5jg","user":"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"}',
// here go 8k lines of this console.log 😅
      } ⭕ An error occurred when validating captcha
      48 |       return await prosopoServer.isVerified(prosopoOutput, maxVerifiedTime || this.maxVerifiedTime);
      49 |     } catch (e) {
    > 50 |       logger.error(`⭕ An error occurred when validating captcha`, e);
         |              ^
      51 |       return false;
      52 |     }
      53 |   }
      at Logger.log (node_modules/@eng-automation/js/src/logger.ts:122:9)
      at Logger.error (node_modules/@eng-automation/js/src/logger.ts:158:12)
      at Procaptcha.validate (src/dripper/Procaptcha.ts:50:14)
      at Object.<anonymous> (src/dripper/Procaptcha.spec.ts:58:20)
  ● Prosopo Procaptcha › Validates captcha positively
    expect(received).toBeTruthy()
    Received: false
      51 |     const procaptcha = new Procaptcha(PROSOPO_SITE_KEY);
      52 |     const result = await procaptcha.validate(PROCAPTCHA_VALID_CAPTCHA);
    > 53 |     expect(result).toBeTruthy();
         |                    ^
      54 |   });
      55 |
      56 |   it("Validates captcha negatively", async () => {
      at Object.<anonymous> (src/dripper/Procaptcha.spec.ts:53:20)
Test Suites: 1 failed, 5 passed, 6 total
Tests:       1 failed, 50 passed, 51 total

and prettier --check has failed with this:

$ prettier ./src ./client/src ./client/tests --check
Checking formatting...
[warn] src/dripper/DripRequestHandler.ts
[warn] src/server/routes/actions.spec.ts
[warn] client/src/app.css
[warn] client/src/app.html
[warn] Code style issues found in 4 files. Forgot to run Prettier?

The error in e2e tests looks papi-related, let's ignore it for now

Copy link
Contributor

@Bullrich Bullrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple of notes, but overall it looks good from my side! Good job!

client/src/lib/components/CaptchaV2.svelte Outdated Show resolved Hide resolved
client/src/lib/components/CaptchaV2.svelte Outdated Show resolved Hide resolved
client/src/lib/components/CaptchaV2.svelte Show resolved Hide resolved
client/tests/faucet.ts Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
src/dripper/DripRequestHandler.ts Outdated Show resolved Hide resolved
src/dripper/polkadot/PolkadotActions.ts Show resolved Hide resolved
forgetso and others added 4 commits April 30, 2024 17:16
* Update paseo RPC wss://sys.dotters.network/paseo (paritytech#398)

* Prettier write

* Make properties public. Remove try catches. Lint

* remove console.log

* Increase test timeout

* try installing playwright with yarn

---------

Co-authored-by: Maksym H <[email protected]>
* Update paseo RPC wss://sys.dotters.network/paseo (paritytech#398)

* Prettier write

* Make properties public. Remove try catches. Lint

* remove console.log

* Increase test timeout

* try installing playwright with yarn

* Moved GitLab checks to GitHub's action (paritytech#402)

Resolves paritytech#401

* Bump yaml from 2.4.1 to 2.4.2 in /client in the npm_and_yarn group across 1 directory (paritytech#403)

* remove empty spaces

* Address more PR comments

* Address comments

* Add type check function to test file

* Fix import so that it works from tests dir2

* Update eng-automation

* fix import

* try after linting

* Suppress ESLint to get CICD to run

* yarn:fix

* pass block number to mock provider

* fix tests

* yarn format:fix

---------

Co-authored-by: Maksym H <[email protected]>
Co-authored-by: Javier Bullrich <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Copy link
Contributor

@Bullrich Bullrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good from my side!

All my concerns have been addressed (with the exception of those two small lint ones)

src/dripper/DripRequestHandler.ts Show resolved Hide resolved
src/server/routes/actions.spec.ts Show resolved Hide resolved
@forgetso
Copy link
Author

forgetso commented May 7, 2024

@Bullrich any idea when this will actually merge into main? I'd like to ensure we are around to support when it's released.

@Bullrich
Copy link
Contributor

Bullrich commented May 7, 2024

@Bullrich any idea when this will actually merge into main? I'd like to ensure we are around to support when it's released.

@mutantcornholio any blockers from your side? Or can we merge?

@Bullrich
Copy link
Contributor

Bullrich commented May 7, 2024

@forgetso FYI the faucet tests are failing: https://github.com/paritytech/polkadot-testnet-faucet/actions/runs/8986761444/job/24684989457?pr=372

I re-run it 3 times and it is still failing 😢

@forgetso
Copy link
Author

forgetso commented May 9, 2024

@forgetso FYI the faucet tests are failing: https://github.com/paritytech/polkadot-testnet-faucet/actions/runs/8986761444/job/24684989457?pr=372

I re-run it 3 times and it is still failing 😢

Hey, sorry! I had it passing in my PR... strange. Will look into this today.

@mordamax
Copy link
Contributor

@forgetso fyi there are some conflicts to resolve :(

@forgetso
Copy link
Author

@forgetso fyi there are some conflicts to resolve :(

Hey @mordamax, thanks for alerting me. I'm out of office until the 27th but will resolve next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client Changes related to th client
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PoC: Migrate to Prosopo
5 participants