-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: add examples, citation and other fixes #34
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
085e6ee
chore(deps): add dev deps necessary for documentation
alexandreteles 956901c
docs: add simple example notebook
alexandreteles 70315e6
docs(fix): fix references to old submodule name
alexandreteles 379fb6e
fix(deps): update dependency setuptools to v75.2.0
renovate[bot] 8568c41
chore(deps): add dev deps necessary for documentation
alexandreteles 1aeabaf
chore: fix deps
alexandreteles 9e3577d
docs: add citation block
alexandreteles 4eeccf0
docs: add citation block
alexandreteles File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,289 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Currently, the TikTok Research API has an issue where, if you have a large number of videos to fetch, you need to make multiple requests to get all the videos and you can only fetch 100 videos for each day. For that reason, we recommend you create a list of dates and make a request for each date. For more information, please check this issue: https://github.com/INCT-DD/tiktok-sdk/issues/27" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from datetime import datetime, timedelta\n", | ||
| "\n", | ||
| "\n", | ||
| "def generate_date_list(start_date: str, end_date: str) -> list[str]:\n", | ||
| " \"\"\"\n", | ||
| " Generate a list of dates between two dates, inclusive.\n", | ||
| "\n", | ||
| " Args:\n", | ||
| " start_date (str): The start date as a string in \"YYYYMMDD\" format.\n", | ||
| " end_date (str): The end date as a string in \"YYYYMMDD\" format.\n", | ||
| "\n", | ||
| " Returns:\n", | ||
| " list of str: A list containing all dates from the start date to the end date,\n", | ||
| " inclusive, each formatted as \"YYYYMMDD\".\n", | ||
| "\n", | ||
| " Raises:\n", | ||
| " ValueError: If the input date strings are not in the correct format or if\n", | ||
| " the start date occurs after the end date.\n", | ||
| "\n", | ||
| " Example:\n", | ||
| " >>> generate_date_list(\"20240811\", \"20240815\")\n", | ||
| " ['20240811', '20240812', '20240813', '20240814', '20240815']\n", | ||
| " \"\"\"\n", | ||
| " date_format = \"%Y%m%d\"\n", | ||
| " start_date = datetime.strptime(start_date, date_format)\n", | ||
| " end_date = datetime.strptime(end_date, date_format)\n", | ||
| " delta = end_date - start_date\n", | ||
| " return [\n", | ||
| " (start_date + timedelta(days=i)).strftime(date_format)\n", | ||
| " for i in range(delta.days + 1)\n", | ||
| " ]\n", | ||
| "\n", | ||
| "\n", | ||
| "start = \"20240816\"\n", | ||
| "end = \"20241003\"\n", | ||
| "\n", | ||
| "dates = generate_date_list(start, end)\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "We recommend you store your client key and secret in the environment variables and call them here." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from os import environ\n", | ||
| "from TikTok.Query import Query\n", | ||
| "from TikTok.Auth import OAuth2\n", | ||
| "from TikTok.ValidationModels.OAuth2 import RequestHeadersModel, TokenRequestBodyModel\n", | ||
| "\n", | ||
| "auth: OAuth2 = await OAuth2.authenticate(\n", | ||
| " headers=RequestHeadersModel(),\n", | ||
| " body=TokenRequestBodyModel(\n", | ||
| " client_key=environ[\"CLIENT_KEY\"],\n", | ||
| " client_secret=environ[\"CLIENT_SECRET\"],\n", | ||
| " ),\n", | ||
| ")\n", | ||
| "\n", | ||
| "query = Query(auth)\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The library uses tenacity to handle retries, but you can also implement your own retry logic as seen below." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from TikTok.ValidationModels.Video import (\n", | ||
| " VideoQueryRequestBuilder,\n", | ||
| " VideoQueryOperation,\n", | ||
| " VideoQueryFieldName,\n", | ||
| " VideoQueryFields,\n", | ||
| " VideoQueryResponseModel,\n", | ||
| ")\n", | ||
| "import asyncio\n", | ||
| "from tqdm import tqdm\n", | ||
| "\n", | ||
| "\n", | ||
| "async def fetch_videos(\n", | ||
| " username: str, date_list: list[str], max_retries: int = 3\n", | ||
| ") -> list[VideoQueryResponseModel]:\n", | ||
| " \"\"\"\n", | ||
| " Fetches videos for a given username over multiple dates, with a progress bar and retry logic.\n", | ||
| "\n", | ||
| " Args:\n", | ||
| " username (str): The TikTok username to query.\n", | ||
| " date_list (list): A list of date strings in \"YYYYMMDD\" format where each date\n", | ||
| " serves as both the start and end date for the query.\n", | ||
| " max_retries (int): Maximum number of retries for failed API calls.\n", | ||
| "\n", | ||
| " Returns:\n", | ||
| " list: A list containing all fetched VideoQueryResponseModel instances.\n", | ||
| " \"\"\"\n", | ||
| " all_videos = []\n", | ||
| "\n", | ||
| " for date in tqdm(date_list, desc=\"Processing Dates\", unit=\"date\"):\n", | ||
| " video_query = VideoQueryRequestBuilder()\n", | ||
| "\n", | ||
| " initial_request = (\n", | ||
| " video_query.start_date(date)\n", | ||
| " .end_date(date)\n", | ||
| " .max_count(100)\n", | ||
| " .and_(VideoQueryOperation.EQ, VideoQueryFieldName.username, [username])\n", | ||
| " .build()\n", | ||
| " )\n", | ||
| "\n", | ||
| " current_request = initial_request.model_copy()\n", | ||
| " has_more = True\n", | ||
| " retries = 0\n", | ||
| "\n", | ||
| " while has_more:\n", | ||
| " try:\n", | ||
| " video_query_response = await query.video.search(\n", | ||
| " request=current_request,\n", | ||
| " fields=[\n", | ||
| " VideoQueryFields.id,\n", | ||
| " VideoQueryFields.video_description,\n", | ||
| " VideoQueryFields.create_time,\n", | ||
| " VideoQueryFields.region_code,\n", | ||
| " VideoQueryFields.share_count,\n", | ||
| " VideoQueryFields.view_count,\n", | ||
| " VideoQueryFields.like_count,\n", | ||
| " VideoQueryFields.comment_count,\n", | ||
| " VideoQueryFields.music_id,\n", | ||
| " VideoQueryFields.hashtag_names,\n", | ||
| " VideoQueryFields.username,\n", | ||
| " VideoQueryFields.effect_ids,\n", | ||
| " VideoQueryFields.playlist_id,\n", | ||
| " VideoQueryFields.favorites_count,\n", | ||
| " ],\n", | ||
| " )\n", | ||
| " except Exception as e:\n", | ||
| " retries += 1\n", | ||
| " if retries > max_retries:\n", | ||
| " break\n", | ||
| " await asyncio.sleep(2**retries) # Exponential backoff\n", | ||
| " continue\n", | ||
| "\n", | ||
| " retries = 0\n", | ||
| "\n", | ||
| " if hasattr(video_query_response.data, \"videos\"):\n", | ||
| " all_videos.extend(video_query_response.data.videos)\n", | ||
| "\n", | ||
| " has_more = getattr(video_query_response.data, \"has_more\", False)\n", | ||
| "\n", | ||
| " if has_more:\n", | ||
| " cursor = getattr(video_query_response.data, \"cursor\", \"\")\n", | ||
| " search_id = getattr(video_query_response.data, \"search_id\", \"\")\n", | ||
| " if not cursor or not search_id:\n", | ||
| " break\n", | ||
| "\n", | ||
| " video_query = VideoQueryRequestBuilder()\n", | ||
| " current_request = (\n", | ||
| " video_query.start_date(date)\n", | ||
| " .end_date(date)\n", | ||
| " .max_count(100)\n", | ||
| " .and_(\n", | ||
| " VideoQueryOperation.EQ, VideoQueryFieldName.username, [username]\n", | ||
| " )\n", | ||
| " .cursor(cursor)\n", | ||
| " .search_id(search_id)\n", | ||
| " .build()\n", | ||
| " )\n", | ||
| "\n", | ||
| " return all_videos\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "If you want to fetch videos from multiple users, you can do it as follows:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "async def fetch_videos_for_usernames(\n", | ||
| " usernames: list[str], date_list: list[str], max_retries: int = 3\n", | ||
| ") -> list[VideoQueryResponseModel]:\n", | ||
| " \"\"\"\n", | ||
| " Fetches videos for multiple usernames over multiple dates, with nested progress bars.\n", | ||
| "\n", | ||
| " Args:\n", | ||
| " usernames (list[str]): A list of TikTok usernames to query.\n", | ||
| " date_list (list[str]): A list of date strings in \"YYYYMMDD\" format where each date\n", | ||
| " serves as both the start and end date for the query.\n", | ||
| " max_retries (int): Maximum number of retries for failed API calls.\n", | ||
| "\n", | ||
| " Returns:\n", | ||
| " list: A list containing all fetched VideoQueryResponseModel instances from all usernames.\n", | ||
| " \"\"\"\n", | ||
| " all_videos = []\n", | ||
| "\n", | ||
| " for username in tqdm(usernames, desc=\"Processing Usernames\", unit=\"username\"):\n", | ||
| " print(f\"Processing: {username}\")\n", | ||
| " user_videos = await fetch_videos(username, date_list, max_retries)\n", | ||
| " all_videos.extend(user_videos)\n", | ||
| "\n", | ||
| " return all_videos\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Remember to replace the list of profiles with your own list of usernames.\n", | ||
| "videos = await fetch_videos_for_usernames(profiles, dates)\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Once you have the list of videos, you can dump it to a dictionary. If you want to load it to a database, you can use the following code:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import pandas as pd\n", | ||
| "\n", | ||
| "df = pd.DataFrame.from_dict(\n", | ||
| " [video.model_dump(by_alias=True, exclude_none=True) for video in videos]\n", | ||
| ")\n", | ||
| "\n", | ||
| "df.fillna(0, inplace=True) # Replace NaN with 0. Optional.\n" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "tiktokresearchapiwrapper-ZMj2YzQb-py3.12", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.12.3" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codebase verification
Incomplete Refactoring: Old Import Path Detected in TikTok/Auth.py
The import statement
from TikTok.Types.OAuth2 import RequestHeadersModel, TokenRequestBodyModelremains inTikTok/Auth.py, indicating that the refactoring toTikTok.ValidationModels.OAuth2is incomplete.TikTok/Auth.py🔗 Analysis chain
Import path modification: Approved with verification recommendation.
The alteration of the import path from
TikTok.Types.OAuth2toTikTok.ValidationModels.OAuth2forRequestHeadersModelandTokenRequestBodyModelis logically sound and maintains consistency with observed changes inTikTok/__init__.py. This modification reflects a deliberate restructuring of the module organization.To ensure comprehensive implementation of this structural change, execute the following verification script:
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
Length of output: 708