diff --git a/README.md b/README.md index 7be37f8..732476c 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # TikTok +[![DOI](https://zenodo.org/badge/852515176.svg)](https://doi.org/10.5281/zenodo.13968316) + TikTok is a Python library for interacting with the TikTok Research API. This library provides a simple and efficient way to access TikTok's API endpoints, @@ -17,7 +19,7 @@ of the Auth class with your API key and secret: ```python from TikTok.Auth import OAuth2 -from TikTok.Types.OAuth2 import RequestHeadersModel, TokenRequestBodyModel +from TikTok.ValidationModels.OAuth2 import RequestHeadersModel, TokenRequestBodyModel auth: OAuth2 = await OAuth2.authenticate( headers=RequestHeadersModel(), @@ -85,3 +87,18 @@ video_query_response = await query.video.search( ``` If you are interested in learning more about the underlying API, you can find the documentation here: [https://developers.tiktok.com/doc/research-api-specs-query-videos](https://developers.tiktok.com/doc/research-api-specs-query-videos) + +# Citation + +If you use this library in your work, please cite it as follows: + +``` +@misc{teles_tiktok_sdk_2024, + author = {Alexandre Teles}, + title = {{TikTok Research API SDK}}, + year = 2024, + doi = {10.5281/zenodo.13968316}, + url = {https://github.com/INCT-DD/tiktok-sdk}, + institution = {Instituto Nacional de Ciência e Tecnologia em Democracia Digital}, +} +``` diff --git a/TikTok/__init__.py b/TikTok/__init__.py index 732eb1c..ffbbaeb 100644 --- a/TikTok/__init__.py +++ b/TikTok/__init__.py @@ -1,4 +1,6 @@ """ +[![DOI](https://zenodo.org/badge/852515176.svg)](https://doi.org/10.5281/zenodo.13968316) + TikTok is a Python library for interacting with the TikTok Research API. This library provides a simple and efficient way to access TikTok's API endpoints, @@ -16,7 +18,7 @@ ```python from TikTok.Auth import OAuth2 -from TikTok.Types.OAuth2 import RequestHeadersModel, TokenRequestBodyModel +from TikTok.ValidationModels.OAuth2 import RequestHeadersModel, TokenRequestBodyModel auth: OAuth2 = await OAuth2.authenticate( headers=RequestHeadersModel(), @@ -85,4 +87,16 @@ If you are interested in learning more about the underlying API, you can find the documentation here: [https://developers.tiktok.com/doc/research-api-specs-query-videos](https://developers.tiktok.com/doc/research-api-specs-query-videos) +And if you use this library in your work, please cite it as follows: + +``` +@misc{teles_tiktok_sdk_2024, + author = {Alexandre Teles}, + title = {{TikTok Research API SDK}}, + year = 2024, + doi = {10.5281/zenodo.13968316}, + url = {https://github.com/INCT-DD/tiktok-sdk}, + institution = {Instituto Nacional de Ciência e Tecnologia em Democracia Digital}, +} +``` """ diff --git a/examples/GetAllUserVideos.ipynb b/examples/GetAllUserVideos.ipynb new file mode 100644 index 0000000..d29c034 --- /dev/null +++ b/examples/GetAllUserVideos.ipynb @@ -0,0 +1,289 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Currently, the TikTok Research API has an issue where, if you have a large number of videos to fetch, you need to make multiple requests to get all the videos and you can only fetch 100 videos for each day. For that reason, we recommend you create a list of dates and make a request for each date. For more information, please check this issue: https://github.com/INCT-DD/tiktok-sdk/issues/27" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timedelta\n", + "\n", + "\n", + "def generate_date_list(start_date: str, end_date: str) -> list[str]:\n", + " \"\"\"\n", + " Generate a list of dates between two dates, inclusive.\n", + "\n", + " Args:\n", + " start_date (str): The start date as a string in \"YYYYMMDD\" format.\n", + " end_date (str): The end date as a string in \"YYYYMMDD\" format.\n", + "\n", + " Returns:\n", + " list of str: A list containing all dates from the start date to the end date,\n", + " inclusive, each formatted as \"YYYYMMDD\".\n", + "\n", + " Raises:\n", + " ValueError: If the input date strings are not in the correct format or if\n", + " the start date occurs after the end date.\n", + "\n", + " Example:\n", + " >>> generate_date_list(\"20240811\", \"20240815\")\n", + " ['20240811', '20240812', '20240813', '20240814', '20240815']\n", + " \"\"\"\n", + " date_format = \"%Y%m%d\"\n", + " start_date = datetime.strptime(start_date, date_format)\n", + " end_date = datetime.strptime(end_date, date_format)\n", + " delta = end_date - start_date\n", + " return [\n", + " (start_date + timedelta(days=i)).strftime(date_format)\n", + " for i in range(delta.days + 1)\n", + " ]\n", + "\n", + "\n", + "start = \"20240816\"\n", + "end = \"20241003\"\n", + "\n", + "dates = generate_date_list(start, end)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We recommend you store your client key and secret in the environment variables and call them here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from os import environ\n", + "from TikTok.Query import Query\n", + "from TikTok.Auth import OAuth2\n", + "from TikTok.ValidationModels.OAuth2 import RequestHeadersModel, TokenRequestBodyModel\n", + "\n", + "auth: OAuth2 = await OAuth2.authenticate(\n", + " headers=RequestHeadersModel(),\n", + " body=TokenRequestBodyModel(\n", + " client_key=environ[\"CLIENT_KEY\"],\n", + " client_secret=environ[\"CLIENT_SECRET\"],\n", + " ),\n", + ")\n", + "\n", + "query = Query(auth)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The library uses tenacity to handle retries, but you can also implement your own retry logic as seen below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from TikTok.ValidationModels.Video import (\n", + " VideoQueryRequestBuilder,\n", + " VideoQueryOperation,\n", + " VideoQueryFieldName,\n", + " VideoQueryFields,\n", + " VideoQueryResponseModel,\n", + ")\n", + "import asyncio\n", + "from tqdm import tqdm\n", + "\n", + "\n", + "async def fetch_videos(\n", + " username: str, date_list: list[str], max_retries: int = 3\n", + ") -> list[VideoQueryResponseModel]:\n", + " \"\"\"\n", + " Fetches videos for a given username over multiple dates, with a progress bar and retry logic.\n", + "\n", + " Args:\n", + " username (str): The TikTok username to query.\n", + " date_list (list): A list of date strings in \"YYYYMMDD\" format where each date\n", + " serves as both the start and end date for the query.\n", + " max_retries (int): Maximum number of retries for failed API calls.\n", + "\n", + " Returns:\n", + " list: A list containing all fetched VideoQueryResponseModel instances.\n", + " \"\"\"\n", + " all_videos = []\n", + "\n", + " for date in tqdm(date_list, desc=\"Processing Dates\", unit=\"date\"):\n", + " video_query = VideoQueryRequestBuilder()\n", + "\n", + " initial_request = (\n", + " video_query.start_date(date)\n", + " .end_date(date)\n", + " .max_count(100)\n", + " .and_(VideoQueryOperation.EQ, VideoQueryFieldName.username, [username])\n", + " .build()\n", + " )\n", + "\n", + " current_request = initial_request.model_copy()\n", + " has_more = True\n", + " retries = 0\n", + "\n", + " while has_more:\n", + " try:\n", + " video_query_response = await query.video.search(\n", + " request=current_request,\n", + " fields=[\n", + " VideoQueryFields.id,\n", + " VideoQueryFields.video_description,\n", + " VideoQueryFields.create_time,\n", + " VideoQueryFields.region_code,\n", + " VideoQueryFields.share_count,\n", + " VideoQueryFields.view_count,\n", + " VideoQueryFields.like_count,\n", + " VideoQueryFields.comment_count,\n", + " VideoQueryFields.music_id,\n", + " VideoQueryFields.hashtag_names,\n", + " VideoQueryFields.username,\n", + " VideoQueryFields.effect_ids,\n", + " VideoQueryFields.playlist_id,\n", + " VideoQueryFields.favorites_count,\n", + " ],\n", + " )\n", + " except Exception as e:\n", + " retries += 1\n", + " if retries > max_retries:\n", + " break\n", + " await asyncio.sleep(2**retries) # Exponential backoff\n", + " continue\n", + "\n", + " retries = 0\n", + "\n", + " if hasattr(video_query_response.data, \"videos\"):\n", + " all_videos.extend(video_query_response.data.videos)\n", + "\n", + " has_more = getattr(video_query_response.data, \"has_more\", False)\n", + "\n", + " if has_more:\n", + " cursor = getattr(video_query_response.data, \"cursor\", \"\")\n", + " search_id = getattr(video_query_response.data, \"search_id\", \"\")\n", + " if not cursor or not search_id:\n", + " break\n", + "\n", + " video_query = VideoQueryRequestBuilder()\n", + " current_request = (\n", + " video_query.start_date(date)\n", + " .end_date(date)\n", + " .max_count(100)\n", + " .and_(\n", + " VideoQueryOperation.EQ, VideoQueryFieldName.username, [username]\n", + " )\n", + " .cursor(cursor)\n", + " .search_id(search_id)\n", + " .build()\n", + " )\n", + "\n", + " return all_videos\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you want to fetch videos from multiple users, you can do it as follows:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "async def fetch_videos_for_usernames(\n", + " usernames: list[str], date_list: list[str], max_retries: int = 3\n", + ") -> list[VideoQueryResponseModel]:\n", + " \"\"\"\n", + " Fetches videos for multiple usernames over multiple dates, with nested progress bars.\n", + "\n", + " Args:\n", + " usernames (list[str]): A list of TikTok usernames to query.\n", + " date_list (list[str]): A list of date strings in \"YYYYMMDD\" format where each date\n", + " serves as both the start and end date for the query.\n", + " max_retries (int): Maximum number of retries for failed API calls.\n", + "\n", + " Returns:\n", + " list: A list containing all fetched VideoQueryResponseModel instances from all usernames.\n", + " \"\"\"\n", + " all_videos = []\n", + "\n", + " for username in tqdm(usernames, desc=\"Processing Usernames\", unit=\"username\"):\n", + " print(f\"Processing: {username}\")\n", + " user_videos = await fetch_videos(username, date_list, max_retries)\n", + " all_videos.extend(user_videos)\n", + "\n", + " return all_videos\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Remember to replace the list of profiles with your own list of usernames.\n", + "videos = await fetch_videos_for_usernames(profiles, dates)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once you have the list of videos, you can dump it to a dictionary. If you want to load it to a database, you can use the following code:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "df = pd.DataFrame.from_dict(\n", + " [video.model_dump(by_alias=True, exclude_none=True) for video in videos]\n", + ")\n", + "\n", + "df.fillna(0, inplace=True) # Replace NaN with 0. Optional.\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "tiktokresearchapiwrapper-ZMj2YzQb-py3.12", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}