- Bypass bot verification

- Impersonate Chrome browser - Resolves #2
Simatwa · Oct 1, 2024 · b27513a · b27513a
1 parent 009afba
commit b27513a
Show file tree

Hide file tree

Showing 10 changed files with 142 additions and 31 deletions.
diff --git a/Docs/CHANGELOG.md b/Docs/CHANGELOG.md
@@ -54,4 +54,10 @@
 
 # v1.0.4
 **What's new?**
-- Bug fixed - **resume download**
+- Bug fixed - **resume download**
+
+# v1.2.0
+**What's new?**
+- Bypass bot verification
+- Impersonate Chrome browser
+- Resolves #2
diff --git a/README.md b/README.md
@@ -62,12 +62,54 @@ cd y2mate-api
 pip install .
 ```
 
-For Windows users you can as well download the executables from [here](https://github.com/Simatwa/y2mate-api/releases/download/v1.0.2/main.exe)
-
 # Usage 
 
 `$ y2mate -f <mp3/mp4> <youtube-link or video id or keyword>`
 
+Following the introduction of Cloudflare protection on the host, y2mate.com, you have to authenticate you're a human right from your **Chrome Browser** and then proceed to extract `cf_clearance` cookie value, which will in turn be passed over to the script.
+
+### `cf_clearance` extraction process.
+
+i. Install **Http Tracker** Chrome extension using [this link](https://chromewebstore.google.com/detail/http-tracker/fklakbbaaknbgcedidhblbnhclijnhbi).
+i. Using **Chrome Browswer**, navigate to y2mate.com and pass the bot verification stage.
+ii. Start the *Http Tracker* extension.
+iii. On the search section of y2mate.com, key-in anything and press <kbd>enter</kbd>
+iv. Return to the Http Tracker window and look for any of the recent urls containing `*y2mate.com*` and click it.
+v. Navigate down to the coookies section and copy the value of key `cf_clearance`. In the proceeding guides I will be referring to this value as **CF-CLEARANCE**
+
+![CF-CLEARANCE highlighted](https://github.com/Simatwa/y2mate-api/blob/main/assets/cf_clearance_highlighted.png?raw=true)
+
+> [!IMPORTANT]
+> Since the script impersonates only one browser, you will have to use **Chrome Browser** for *CF-CLEARANCE* extraction process.
+
+The uncompromised usage will be:
+
+`$ y2mate -f <mp3/mp4> -cf <CF-CLEARANCE> <youtube-link or video id or keyword>`
+
+
+To developers:
+
+```python
+from y2mate_api import session
+
+session.cookies.update(
+  {
+    "cf_clearance": cf_clearance_value
+  }
+)
+```
+
+As a workaround to explicitly declaring the CF-CLEARANCE along with other commands, export it to the environment as `Y2MATE_CF_CLEARANCE`.
+
+By doing that, something like this will still get the work done:
+
+`$ y2mate -f <mp3/mp4> <youtube-link or video id or keyword>`
+
+> [!WARNING]
+> The CF-CLEARANCE expires after a short while, consider updating it more frequently.
+
+
+
 <details>
 
 <summary>
@@ -205,10 +247,10 @@ For more info run `$ y2mate -h`
 ```
 usage: y2mate [-h] [-v] [-f mp3|mp4]
               [-q 4k|1080p|720p|480p|360p|240p|144p|auto|best|worst|mp3|m4a|.m4a|128kbps|192kbps|328kbps]
-              [-r m4a|3gp|mp4|mp3] [-k [KEYWORD ...]] [-a [AUTHOR ...]]
-              [-l LIMIT] [-d PATH] [-t TIMEOUT] [-c CHUNK] [-i PATH]
-              [-o FORMAT] [-thr THREAD] [--disable-bar] [--confirm] [--unique]
-              [--quiet] [--history] [--clear] [--resume] [--play]
+              [-r m4a|3gp|mp4|mp3] [-k [KEYWORD ...]] [-a [AUTHOR ...]] [-l LIMIT]
+              [-d PATH] [-t TIMEOUT] [-c CHUNK] [-i PATH] [-o FORMAT] [-cf COOKIE]
+              [-thr THREAD] [--disable-bar] [--confirm] [--unique] [--quiet] [--history]
+              [--clear] [--resume] [--play]
               [query ...]
 
 Download youtube videos and audios by title or link
@@ -224,16 +266,15 @@ options:
   -q 4k|1080p|720p|480p|360p|240p|144p|auto|best|worst|mp3|m4a|.m4a|128kbps|192kbps|328kbps, --quality 4k|1080p|720p|480p|360p|240p|144p|auto|best|worst|mp3|m4a|.m4a|128kbps|192kbps|328kbps
                         Media quality - auto
   -r m4a|3gp|mp4|mp3, --resolver m4a|3gp|mp4|mp3
-                        Other media formats incase of multiple options -
-                        mp4/mp3
+                        Other media formats incase of multiple options - mp4/mp3
   -k [KEYWORD ...], --keyword [KEYWORD ...]
                         Media should contain this keywords - None
   -a [AUTHOR ...], --author [AUTHOR ...]
                         Media author i.e YouTube channel name - None
   -l LIMIT, --limit LIMIT
                         Total videos to be downloaded - 1
   -d PATH, --dir PATH   Directory for saving the contents -
-                        /home/smartwa/git/Smartwa/I-learn-this
+                        /home/smartwa/git/smartwa/y2mate-api
   -t TIMEOUT, --timeout TIMEOUT
                         Http request timeout in seconds - 30
   -c CHUNK, --chunk CHUNK
@@ -243,6 +284,8 @@ options:
   -o FORMAT, --output FORMAT
                         Format for generating filename %(key)s :
                         [title,vid,fquality,ftype] or 'pretty' - None
+  -cf COOKIE, --cf-clearance COOKIE
+                        cf_clearance cookie value for y2mate.com
   -thr THREAD, --thread THREAD
                         Download [x] amount of videos/audios at once - 1
   --disable-bar         Disable download progress bar - False

diff --git a/assets/cf_clearance_highlighted.png b/assets/cf_clearance_highlighted.png
diff --git a/requirements.txt b/requirements.txt
@@ -5,3 +5,4 @@ colorama==0.4.6
 appdirs==1.4.4
 click==8.1.3
 brotli==1.1.0
+curl_cffi==0.7.2
diff --git a/setup.py b/setup.py
@@ -1,6 +1,6 @@
 from setuptools import setup
 
-version = "1.0.6"
+version = "1.2.0"
 info = "Download youtube videos and audios by title or link"
 author = "Smartwa"
 repo = "https://github.com/Simatwa/y2mate-api"
@@ -24,8 +24,9 @@
         "appdirs==1.4.4",
         "click==8.1.3",
         "brotli==1.1.0",
+        "curl_cffi==0.7.2",
     ],
-    python_requires=">=3.8",
+    python_requires=">=3.9",
     long_description=open("README.md").read(),
     long_description_content_type="text/markdown",
     classifiers=[
@@ -42,7 +43,7 @@
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
         "Programming Language :: Python :: 3.11",
-        "Programming Language :: Python :: 3.12"
+        "Programming Language :: Python :: 3.12",
     ],
     entry_points={
         "console_scripts": [

diff --git a/test.py b/test.py
@@ -0,0 +1,41 @@
+# import requests
+
+from curl_cffi import requests
+
+headers = {
+    "Accept": "*/*",
+    "Accept-Encoding": "gzip, deflate, br, zstd",
+    "Accept-Language": "en-US,en;q=0.5",
+    # "Connection":	"keep-alive",
+    # "Content-Type"	:"application/x-www-form-urlencoded; charset=UTF-8",
+    # "Origin":	"https://www.y2mate.com",
+    "Referer": "https://www.y2mate.com/search/hello",
+    # "Upgrade-Insecure-Requests":	"1",
+    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
+    # "X-Requested-With":	"XMLHttpRequest",
+}
+
+payload = {
+    "k_query": "https://youtu.be/akEZeHBOyko?si=3p2WreYh6-3EoIpc",
+    "k_page": "home",
+    "hl": "en",
+    "q_auto": "0",
+}
+
+session = requests.Session()
+session.headers.update(headers)
+session.cookies.update(
+    {
+        "cf_clearance": "0bKyUQFwyXTJ9IdaSCfUZ2_YsZtP9WpeCnjb2UXZZhs-1727812278-1.2.1.1-Almns7kQgsfC4S8oy08yZMvhZXZOa5M1kTxTMSunnnSoqT1qruuGUUzx0M_gnd2ps2qoJ0rOoL3k0TASKfA3dP4HPbFI11S9lNYvP0IjcieG3yFRqA5TSN0wVfJRkvYxMwPCUNVVTz9oAy9ARRGy_IyMNeQu2.NgSFUFJYD0h4v.DA0hwVwgrz4e2Kf5rjFp5ZCYZHMasuUZcnBMOVp9pQE63lBT6l7k0H902pFHvpqE7Y2Y19wHGeQq2FcywYSITB_QzyME2t1tLC1m53MvFrBnxfH5SQ2iIZF1hL7s4FihnPJQs32ik8kAm.TyewgT3MXs2xq1B_vIEAsvbSzQqjZ8scMrIeRppBwfL59JHvu23JkwwN_kqvK2mH1DhVQQ3FvWVbSerNqwjQjUll_iO5fBn00LzuScBep1aRLOHlI"
+    }
+)
+
+
+def main():
+    url = "https://www.y2mate.com/mates/en948/analyzeV2/ajax"
+    resp = session.post(url, data=payload, impersonate="chrome")
+    print(resp.status_code, resp.reason)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/y2mate_api/__init__.py b/y2mate_api/__init__.py
@@ -9,12 +9,7 @@
 from .main import second_query
 from .main import third_query
 from .main import appdir
+from .main import session
 from .downloader import Handler
 
-__all__ = [
-    "first_query",
-    "second_query",
-    "third_query",
-    "Handler",
-    "appdir"
-]
+__all__ = ["first_query", "second_query", "third_query", "Handler", "appdir", "session"]
diff --git a/y2mate_api/console.py b/y2mate_api/console.py
@@ -2,7 +2,7 @@
 import logging
 from . import __version__, __info__, __disclaimer__
 from .main import utils
-from os import getcwd, remove
+from os import getcwd, remove, getenv
 from sys import exit
 from .main import history_path, utils
 
@@ -112,6 +112,12 @@ def get_args():
         metavar="FORMAT",
         help="Format for generating filename %%(key)s : [title,vid,fquality,ftype] or 'pretty' - %(default)s",
     )
+    parser.add_argument(
+        "-cf",
+        "--cf-clearance",
+        help="cf_clearance cookie value for y2mate.com",
+        metavar="COOKIE",
+    )
     parser.add_argument(
         "-thr",
         "--thread",
@@ -185,9 +191,11 @@ def main():
         dir=args.dir,
         progress_bar=args.disable_bar == False,
         quiet=args.quiet,
-        naming_format=f"%(title)s{' - %(fquality)s' if args.format=='mp4' else ''}.%(ftype)s"
-        if str(args.output).lower() == "pretty"
-        else args.output,
+        naming_format=(
+            f"%(title)s{' - %(fquality)s' if args.format=='mp4' else ''}.%(ftype)s"
+            if str(args.output).lower() == "pretty"
+            else args.output
+        ),
         chunk_size=args.chunk,
         play=args.play,
         format=args.format,
@@ -198,6 +206,11 @@ def main():
         author=h_mult_args(args.author),
         resume=args.resume,
     )
+    cf_clearance_value = args.cf_clearance or getenv("Y2MATE_CF_CLEARANCE")
+    if cf_clearance_value:
+        from . import session
+
+        session.cookies.update({"cf_clearance": cf_clearance_value})
     logging.info(f"y2mate launched - v{__version__}")
     if args.input:
         for query in open(args.input).read().strip().split("\n"):

diff --git a/y2mate_api/downloader.py b/y2mate_api/downloader.py
@@ -15,6 +15,7 @@
 from sys import stdout
 from click import launch as launch_media, confirm as confirm_from_user
 import warnings
+import requests as requests_native
 
 """
 - query string
@@ -402,7 +403,9 @@ def save(
                     current_downloaded_size / 1000000, 2
                 )  # convert to mb
 
-            resp = requests.get(third_dict["dlink"], stream=True, headers=mod_headers)
+            resp = requests_native.get(
+                third_dict["dlink"], stream=True, headers=mod_headers
+            )
 
             default_content_length = 0
             size_in_bytes = int(
@@ -433,8 +436,8 @@ def save(
                 if any([save_to.startswith("/"), ":" in save_to])
                 else path.join(getcwd(), dir, filename)
             )
-            try_play_media = (
-                lambda: launch_media(third_dict["saved_to"]) if play else None
+            try_play_media = lambda: (
+                launch_media(third_dict["saved_to"]) if play else None
             )
             saving_mode = "ab" if resume else "wb"
             if progress_bar:

diff --git a/y2mate_api/main.py b/y2mate_api/main.py
@@ -1,4 +1,5 @@
-import requests
+# import requests
+from curl_cffi import requests
 import logging
 from time import sleep
 import json
@@ -8,11 +9,11 @@
 from sys import exit
 
 __prog__ = "y2mate"
-session = requests.session()
+session = requests.Session()
 
 headers = {
     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
-    "User-Agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36",
+    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
     "Accept-Encoding": "gzip, deflate, br",
     "Accept-Language": "en-US,en;q=0.9",
     "referer": "https://y2mate.com",
@@ -67,12 +68,14 @@ def main(*args, **kwargs):
     @staticmethod
     def get(*args, **kwargs):
         r"""Sends http get request"""
+        kwargs["impersonate"] = "chrome"
         resp = session.get(*args, **kwargs)
         return all([resp.ok, "application/json" in resp.headers["content-type"]]), resp
 
     @staticmethod
     def post(*args, **kwargs):
         r"""Sends http post request"""
+        kwargs["impersonate"] = "chrome"
         resp = session.post(*args, **kwargs)
         return all([resp.ok, "application/json" in resp.headers["content-type"]]), resp
 
@@ -183,7 +186,11 @@ def main(self, timeout=30):
             self.processed = True
         else:
             logging.debug(f"{resp.headers.get('content-type')} - {resp.content}")
-            logging.error(f"First query failed - [{resp.status_code} : {resp.reason}")
+            logging.error(f"First query failed - [{resp.status_code} : {resp.reason}]")
+            if session.cookies.get('cf_clearance'):
+                logging.info('Seems like CF-CLEARANCE cookie has expired!')
+            else:
+                logging.info('Try passing CF-CLEARANCE cookie.')
         return self
 
 
@@ -322,6 +329,7 @@ def __init__(self, query_two: object):
         self.qualities = {
             self.formats[0]: [
                 "4k",
+                "2k",
                 "1080p",
                 "720p",
                 "480p",