-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #299 from apivideo/Add-GCS-migration-guide
Add GCS migration guide
- Loading branch information
Showing
9 changed files
with
40 additions
and
174 deletions.
There are no files selected for viewing
Binary file added
BIN
+28.5 KB
...lates/documentation/_assets/get-started/migration-guide/Google-Storage-Logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 0 additions & 1 deletion
1
...lates/documentation/_assets/get-started/migration-guide/Google-Storage-Logo.svg
This file was deleted.
Oops, something went wrong.
Binary file added
BIN
+65.1 KB
templates/documentation/_assets/get-started/migration-guide/gcs-migration-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+195 KB
templates/documentation/_assets/get-started/migration-guide/gcs-migration-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+116 KB
templates/documentation/_assets/get-started/migration-guide/gcs-migration-3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+322 KB
templates/documentation/_assets/get-started/migration-guide/gcs-migration-4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,213 +1,76 @@ | ||
--- | ||
title: "Amazon S3 migration guide" | ||
slug: "aws-migration" | ||
title: "Google Cloud Storage migration guide" | ||
slug: "gcs-migration" | ||
meta: | ||
description: This page gets users started on how to migrate from Amazon S3 to api.video. | ||
description: This page gets users started on how to migrate from Google Cloud Storage to api.video. | ||
--- | ||
|
||
# Amazon S3 migration guide | ||
# Google Cloud Storage migration guide | ||
|
||
Planning to migrate api.video from Amazon S3? We got you covered! | ||
Planning to migrate api.video from Google Cloud Storage? We got you covered! | ||
|
||
There are two methods at your disposal that you can use to migrate all of your video content from S3 to api.video. | ||
Using the api.video Import tool, it only takes a couple of clicks to migrate all of your video content from Google Cloud Storage to api.video. This simple tool only requires read access to your Google Cloud Storage bucket, and that you authenticate yourself with your api.video account. | ||
|
||
- Migrate with api.video Import tool | ||
- Migrate using your own script | ||
Check out the [Import Tool](https://import.api.video/) to get started. | ||
|
||
## What's the cost? | ||
|
||
We understand that when you want to move to a different provider, it takes effort and development resources. You also want to make sure that it is cost-efficient, especially if you are moving to api.video to save costs. | ||
|
||
api.video gives you the ability to **migrate for free** and avoid paying anything for encoding! | ||
|
||
## api.video Import Tool | ||
## Granting read access to your GCS bucket | ||
|
||
With api.video, you can use our in-house import tool to migrate all of your videos in a short time from the popular hosting and video provider. | ||
To access your content on Google Cloud Storage, api.video needs read access. This means that you need to grant the Import Tool's external service account read access to GCS objects. | ||
|
||
The simple tool will require you to input several parameters from the provider you are leaving and from api.video. | ||
You can grant this access in two quick steps: | ||
|
||
Check out the [Import Tool](https://import.api.video/) now. | ||
* select the project and bucket that you want to migrate | ||
* grant read access to the Import Tool | ||
|
||
### How to get the AWS Access Key and Account Secret | ||
### 1. Select your project and bucket | ||
|
||
To access AWS, you will need to sign up for an AWS account. | ||
* Go to the [Google Cloud Console](https://console.cloud.google.com/), log in to your account and select the project you want to use. | ||
* Navigate to Cloud Storage > Buckets. | ||
|
||
Access keys consist of an access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. If you don't have access keys, you can create them by using the [IAM console](https://console.aws.amazon.com/iam/). | ||
![Showing the Buckets menu in Google Cloud Storage](/_assets/get-started/migration-guide/gcs-migration-1.png) | ||
|
||
We recommend that you use IAM access keys instead of AWS root account access keys. IAM lets you securely control access to AWS services and resources in your AWS account. | ||
* Select the bucket that you want to migrate. | ||
|
||
{% capture content %} | ||
![Showing the list of buckets in Google Cloud Storage](/_assets/get-started/migration-guide/gcs-migration-2.png) | ||
|
||
To create access keys, you must have permission to perform the required IAM actions. For more information, see Granting IAM User Permission to Manage Password Policy and Credentials in the IAM User Guide. | ||
* Click on the "Permissions" tab and click on "Grant access". This enables you to add a new member to the bucket. | ||
|
||
{% endcapture %} | ||
![Showing the Grant access button in Google Cloud Storage](/_assets/get-started/migration-guide/gcs-migration-3.png) | ||
|
||
{% include "_partials/callout.html" kind: "info", content: content %} | ||
### 2. Grant read access to the Import Tool | ||
|
||
**To get your access key ID and secret access key:** | ||
- Open the [IAM console](https://console.aws.amazon.com/iam/). | ||
- On the navigation menu, choose Users. | ||
- Choose your IAM user name (not the check box). | ||
- Open the Security credentials tab, and then choose Create access key. | ||
- To see the new access key, choose Show. Your credentials resemble the following: | ||
Once you have navigated to the bucket that you want to migrate, you need to add the Import Tool as a principal and assign Storage Object Viewer role: | ||
|
||
``` | ||
![Showing the Grant access dialog box Google Cloud Storage](/_assets/get-started/migration-guide/gcs-migration-3.png) | ||
|
||
Access key ID: AKIAIOSFODNN7EXAMPLE | ||
* In the "Grant access" dialog box, enter `storage-service-account@video-import-tool.iam.gserviceaccount.com` in the "New principals" input field. | ||
* In the "Role" dropdown menu, select "Storage Object Viewer". | ||
* Save your changes. | ||
|
||
Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY | ||
The Import Tool's service account now has read access to the Google Cloud Storage bucket you selected. You can now import videos from this bucket. | ||
|
||
``` | ||
## Import the videos | ||
|
||
- To download the key pair, choose Download .csv file. Store the .csv file with keys in a secure location. | ||
|
||
{% capture content %} | ||
|
||
Keep the keys confidential to protect your AWS account, and never email them. Do not share them outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key. | ||
|
||
{% endcapture %} | ||
|
||
{% include "_partials/callout.html" kind: "warning", content: content %} | ||
|
||
You can retrieve the secret access key only when you initially create the key pair. Like a password, **you can't retrieve it later.** If you lose it, you must create a new key pair. | ||
|
||
### Importing the videos | ||
|
||
Once you have all the credentials, you can proceed with the migration. | ||
Once you have granted all necessary access, you can start the migration. | ||
|
||
1. Navigate to the api.video [Import Tool](https://import.api.video/). | ||
|
||
2. Select Amazon S3 from the list. | ||
|
||
3. Autherize with api.video with your account. This process will link your api.video workspace to the Import tool. | ||
|
||
4. Select the project that you would like to import the video. | ||
|
||
5. Next, click on Sign in to Amazon S3 and fill in the AWS Access Key and AWS Secret Key you got from Amazon S3. | ||
|
||
6. Select the S3 bucket you would like to import. | ||
|
||
7. Now you can proceed with the import. | ||
|
||
The process will show you the status of each video and the encoding status. | ||
|
||
## Importing with a script | ||
|
||
Another way to import your videos is to use your own script and leverage one of the api.video client libraries with the provider's client libraries or API (if there are any). | ||
|
||
### Setting up | ||
|
||
What you'll need to run the script: | ||
|
||
- api.video API key | ||
- AWS credentials | ||
- Node.js | ||
|
||
### Getting the api.video API key | ||
|
||
In order to get your API key. Navigate to the [api.video dashboard](https://dashboard.api.video/videos) (if you are not already signed up, sign up with api.video) and continue to copy your API key from the **Overview page** or the **API Keys page** | ||
|
||
### What the script will do? | ||
|
||
The below script is designed to run through all of the files in a specific S3 bucket, generate a signed URL, and pass it on to api.video as the source. api.video will then take the URL and ingest the video directly from S3. | ||
|
||
Make sure that the script is modified according to your needs. | ||
|
||
### Preparation | ||
|
||
If you are new to Node.js, here's how you can run this script. | ||
|
||
Make sure that you have Node.js installed, if it doesn't exist on your machine, you can install it by following this [guide](https://kinsta.com/blog/how-to-install-node-js/). | ||
|
||
Now that you have Node.js installed, we will have to create the path of the script and install the dependencies. | ||
|
||
1. Create the folder for the script: | ||
```shell | ||
$ mkdir s3-apivideo-migration | ||
``` | ||
|
||
2. Navigate to the folder | ||
```shell | ||
$ cd s3-apivideo-migration | ||
``` | ||
|
||
3. Initialize `npm` in order to create the package.json file for the module dependacies | ||
```shell | ||
$ npm init | ||
``` | ||
|
||
4. Install the modules that we will need to run | ||
```shell | ||
$ npm install @aws-sdk/client-s3 --save | ||
$ npm install @aws-sdk/s3-request-presigner --save | ||
$ npm install @api.video/nodejs-client --save | ||
``` | ||
|
||
5. Now create an index file that you would like the script to exist in: | ||
```shell | ||
$ touch index.js | ||
``` | ||
|
||
6. Copy over the below script and replace the following parameters with your own: `videBucket`, `region`, `accessKeyId`, `secrectAccessKey`, `apivideoCreds` | ||
|
||
```javascript | ||
import { S3Client, ListObjectsV2Command, GetObjectCommand, S3 } from "@aws-sdk/client-s3"; | ||
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; | ||
import ApiVideoClient from '@api.video/nodejs-client'; | ||
|
||
// your S3 bucket name | ||
const videoBucket = "videos-cold-storage-api-video"; | ||
// your S3 bucket region | ||
const region = 'eu-north-1'; | ||
// your S3 credentails | ||
const s3creds = { | ||
accessKeyId: 'XXXXXX', | ||
secretAccessKey: 'XXXXXXX', | ||
} | ||
// api.video api key | ||
const apivideoCreds = 'XXXXXXX'; | ||
2. Select **Google Cloud Storage** from the list. | ||
|
||
3. Log into your api.video account. This process links your api.video workspace to the Import tool. | ||
|
||
const apivideoClient = new ApiVideoClient({ apiKey: apivideoCreds }); | ||
4. If you have multiple api.video projects in your workspace, select the project where you want to migrate your content to. | ||
|
||
const client = new S3Client({ region, | ||
credentials: s3creds | ||
}); | ||
5. Next, enter the name of the Google Cloud Storage bucket that you want to migrate. | ||
|
||
new S3({ region, | ||
credentials: s3creds | ||
}); | ||
6. Next, the tool retrieves the available videos from Google Cloud Storage. Select the videos you would like to import. | ||
|
||
const bucketName = { | ||
Bucket: videoBucket, | ||
}; | ||
const listBucketObjectsCommand = new ListObjectsV2Command(bucketName); | ||
7. Select the videos you want to migrate, and start the import process. | ||
|
||
try { | ||
const listObjects = await client.send(listBucketObjectsCommand); | ||
if (listObjects?.$metadata?.httpStatusCode === 200 && listObjects?.Contents.length > 0) { | ||
for (let i in listObjects.Contents) { | ||
const objectKey = listObjects.Contents[i].Key; | ||
bucketName.Key = objectKey; | ||
const objectDetailsCommand = new GetObjectCommand(bucketName); | ||
const objectDetails = await client.send(objectDetailsCommand); | ||
if(/video/.test(objectDetails.ContentType)) { | ||
const signedUrl = await getSignedUrl(client, objectDetailsCommand, { expiresIn: 3600 }); | ||
const videoCreationPayload = { | ||
title: objectKey, | ||
source: signedUrl | ||
}; | ||
await apivideoClient.videos.create(videoCreationPayload); | ||
console.log("upload complete") | ||
} | ||
} | ||
} | ||
} catch (e) { | ||
console.error(e); | ||
} | ||
``` | ||
7. Save the file and run the script by: | ||
```shell | ||
$ node index.js | ||
``` | ||
The process displays the upload status and encoding status for each video. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters