Merge branch 'feature/update_ffmpeg' into develop

Author: koxon

Dockerfile

@@ -1,4 +1,4 @@
-FROM sportarc/cloudtranscode-base:3.3
+FROM sportarc/cloudtranscode-base:4.2
 MAINTAINER bFAN Sports
 
 COPY . /usr/src/cloudtranscode

README.md

@@ -1,8 +1,8 @@
 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/sportarchive/CloudTranscode?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 [![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/sportarchive/CloudTranscode/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/sportarchive/CloudTranscode/?branch=master)
 
-### Updates 2018/10/17
-Cleaning up the doc. Fixed an issue. Trying to make the documentation more accessible.
+### Updates 2020/05/09
+Update to FFMpeg 4.2
 
 # What is Cloud Transcode?
 Cloud Transcode (CT) is your own distributed transcoding stack. With it you can transcode media files in a distributed way, at scale.
@@ -42,13 +42,13 @@ Those Activities listen to the Amazon SFN (Step functions) service for incoming
 Tasks and Workflows (aka `State Machine`) are defined in the AWS SFN console, and are identified by their AWS ARNs (AWS resources identifier). You can then start `N` activity workers that will start listening for incoming jobs and execute them.
 
 You can scale your infrastructure in AWS based on your volume, capacity, cost, resources, etc.
-You can run those SFN Activities on Docker, which is recommended. A Dockerfile is provided for you. 
+You can run those SFN Activities on Docker, which is recommended. A Dockerfile is provided for you.
 But you can run on anything and anywhere.
 
 Your client applications can initiate new SFN workflows using the AWS SDK of your choice. The client apps will pass JSON input date to AWS SFN. <br>
 SFN will then pass this input to your activities, which will then return a JSON output. This output can be passed on to the next activities.
 
-Thanks to the `CloudProcessingEngine-SDK` you can build your own workflow and call any activities you want. You can implement your own activities as well. 
+Thanks to the `CloudProcessingEngine-SDK` you can build your own workflow and call any activities you want. You can implement your own activities as well.
 
 Cloud Transcode could use help on the following Activities if you are interested in participating:
 
@@ -87,16 +87,16 @@ One TranscodeAssetActivity worker processes all outputs wanted, in sequence, not
 
 Activities are standalone scripts writen in PHP (legacy reasons, but it's clean!) that can be started in command line.
 
-``` 
+```
 bash $> ./src/activities/ValidateAssetActivity.php -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:ValidateAsset
 bash $> ./src/activities/TranscodeAssetActivity.php -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeAsset
 ```
 
 Or using Docker
 
 ```
-$> sudo docker run sportarc/cloudtranscode ValidateAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:ValidateAsset
-$> sudo docker run sportarc/cloudtranscode TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeAsset
+$> sudo docker run sportarc/cloudtranscode:4.2 ValidateAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:ValidateAsset
+$> sudo docker run sportarc/cloudtranscode:4.2 TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeAsset
 ```
 
 Using these commands, you can start an activity worker that processes one type of activity. In these cases `ValidateAssetActivity` and `TranscodeAssetActivity`
@@ -128,7 +128,7 @@ A Dockerfile like this for example:
 
 
 ``` Dockerfile
-FROM sportarc/cloudtranscode:3.3
+FROM sportarc/cloudtranscode:4.2
 MAINTAINER bFAN Sports
 
 COPY clientInterfaces /usr/src/clientInterfaces
@@ -140,10 +140,10 @@ Then build your own image as follow: `sudo docker build -t sportarc/cloudtransco
 Then you can start your workers like this:
 
 ```
-$> sudo docker run sportarc/cloudtranscode-prod ValidateAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:ValidateAsset -C /usr/src/clientInterfaces/ValidateAssetClientInterfaces.php
-$> sudo docker run sportarc/cloudtranscode-prod TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeAllOutputAssets -C /usr/src/clientInterfaces/TranscodeAllOutputAssetsClientInterfaces.php
-$> sudo docker run sportarc/cloudtranscode-prod TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeImageAsset -C /usr/src/clientInterfaces/TranscodeImagesAssetsClientInterfaces.php
-$> sudo docker run sportarc/cloudtranscode-prod TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:OnDemandTranscodeAsset -C /usr/src/clientInterfaces/OnDemandTranscodeAssetClientInterfaces.php
+$> sudo docker run sportarc/cloudtranscode-prod:4.2 ValidateAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:ValidateAsset -C /usr/src/clientInterfaces/ValidateAssetClientInterfaces.php
+$> sudo docker run sportarc/cloudtranscode-prod:4.2 TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeAllOutputAssets -C /usr/src/clientInterfaces/TranscodeAllOutputAssetsClientInterfaces.php
+$> sudo docker run sportarc/cloudtranscode-prod:4.2 TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:TranscodeImageAsset -C /usr/src/clientInterfaces/TranscodeImagesAssetsClientInterfaces.php
+$> sudo docker run sportarc/cloudtranscode-prod:4.2 TranscodeAssetActivity -A arn:aws:states:eu-west-1:XXXXXXXXXXXX:activity:OnDemandTranscodeAsset -C /usr/src/clientInterfaces/OnDemandTranscodeAssetClientInterfaces.php
 ```
 
 As you can see, you can create many SFN tasks. Each task will execute the same activity code, but they are connected to different client applications using different Interface classes.
@@ -227,12 +227,12 @@ Thanks for contributing !
 
 # FFmpeg
 
-CloudTranscode uses FFmpeg 3.3
+CloudTranscode uses FFmpeg 4.2
 
 The CloudTranscode Docker image is based on two other images:
 
-   - https://hub.docker.com/r/sportarc/ffmpeg/: Base image containing: Ubuntu 14, PHP CLI 5.6, FFmpeg
-   - https://hub.docker.com/r/sportarc/cloudtranscode-base/: Adds `ImageMagic` to the above image
+   - https://hub.docker.com/r/sportarc/ffmpeg/
+   - https://hub.docker.com/r/sportarc/cloudtranscode-base/
 
 
 # FFMpeg performance benchmark on Amazon EC2

src/activities/ValidateAssetActivity.php

@@ -3,7 +3,7 @@
 <?php
 
 /*
- *   This class validate input assets and get mime type and metadata 
+ *   This class validate input assets and get mime type and metadata
  *   Using the ValidateAsset activity you can confirm your asset can be transcoded
  *
  *   Copyright (C) 2016  BFan Sports - Sport Archive Inc.
@@ -32,15 +32,17 @@ class ValidateAssetActivity extends BasicActivity
     private $finfo;
     private $s3;
     private $curl_data = '';
-    
+
+    const VALIDATE_ASSET_FAILED = "VALIDATE_ASSET_FAILED";
+
     public function __construct($client = null, $params, $debug, $cpeLogger)
     {
         # Check if preper env vars are setup
         if (!($region = getenv("AWS_DEFAULT_REGION")))
             throw new CpeSdk\CpeException("Set 'AWS_DEFAULT_REGION' environment variable!");
-        
+
         parent::__construct($client, $params, $debug, $cpeLogger);
-        
+
         $this->s3 = new \Aws\S3\S3Client([
             "version" => "latest",
             "region"  => $region
@@ -51,17 +53,17 @@ public function __construct($client = null, $params, $debug, $cpeLogger)
     private function writefn($ch, $chunk)
     {
         static $limit = 1024; // 500 bytes, it's only a test
-        
+
         $len = strlen($this->curl_data) + strlen($chunk);
         if ($len >= $limit ) {
             $this->curl_data .= substr($chunk, 0, $limit-strlen($this->curl_data));
             return -1;
         }
-        
+
         $this->curl_data .= $chunk;
         return strlen($chunk);
     }
-    
+
     // Perform the activity
     public function process($task)
     {
@@ -74,21 +76,44 @@ public function process($task)
 
         // Call parent process:
         parent::process($task);
-        
+
         $this->activityHeartbeat();
         $tmpFile = tempnam(sys_get_temp_dir(), 'ct');
 
         if (isset($this->input->{'input_asset'}->{'http'})) {
+            $this->cpeLogger->logOut(
+                "DEBUG",
+                basename(__FILE__),
+                "Downloading first 1024 bytes from: " . $this->input->{'input_asset'}->{'http'},
+                $this->logKey
+            );
+
             $ch = curl_init();
             curl_setopt($ch, CURLOPT_URL, $this->input->{'input_asset'}->{'http'});
             curl_setopt($ch, CURLOPT_RANGE, '0-1024');
             curl_setopt($ch, CURLOPT_WRITEFUNCTION, array($this, 'writefn'));
             curl_exec($ch);
             curl_close($ch);
+
+            if ($errno = curl_errno($ch)) {
+                $error_message = curl_strerror($errno);
+                throw new CpeSdk\CpeException(
+                    $error_message,
+                    self::VALIDATE_ASSET_FAILED
+                );
+            }
+
             $chunk = $this->curl_data;
         }
         else if (isset($this->input->{'input_asset'}->{'bucket'}) &&
                  isset($this->input->{'input_asset'}->{'file'})) {
+            $this->cpeLogger->logOut(
+                "DEBUG",
+                basename(__FILE__),
+                "Downloading first 1024 bytes from S3. Bucket: " . $this->input->{'input_asset'}->{'bucket'} . ". Key: " . $this->input->{'input_asset'}->{'file'},
+                $this->logKey
+            );
+
             // Fetch first 1 KiB of the file for Magic number validation
             $obj = $this->s3->getObject([
                 'Bucket' => $this->input->{'input_asset'}->{'bucket'},
@@ -97,7 +122,7 @@ public function process($task)
             ]);
             $chunk = (string) $obj['Body'];
         }
-        
+
         $this->activityHeartbeat();
 
         // Determine file type
@@ -112,7 +137,7 @@ public function process($task)
                 basename(__FILE__),
                 "File meta information gathered. Mime: $mime | Type: $type",
                 $this->logKey
-        );
+            );
 
         // Load the right transcoder base on input_type
         // Get asset detailed info
@@ -147,14 +172,14 @@ public function process($task)
                 }
             }
         }
-        
+
         $assetInfo->mime = $mime;
         $assetInfo->type = $type;
 
         $result['input_asset']     = $this->input->{'input_asset'};
         $result['input_metadata']  = $assetInfo;
         $result['output_assets']   = $this->input->{'output_assets'};
-        
+
         return json_encode($result);
     }
 }
@@ -188,11 +213,11 @@ function check_activity_arguments()
     global $debug;
     global $clientClassPath;
     global $name;
-    
+
     // Handle input parameters
     if (!($options = getopt("N:A:l:C:hd")))
         usage();
-    
+
     if (isset($options['h']))
         usage();
 
@@ -216,7 +241,7 @@ function check_activity_arguments()
     if (isset($options['N']) && $options['N']) {
         $name = $options['N'];
     }
-    
+
     if (isset($options['l']))
         $logPath = $options['l'];
 }
@@ -251,4 +276,4 @@ function check_activity_arguments()
 
 // Initiate the polling loop and will call your `process` function upon trigger
 // The process will exit after 24 hours (1440 minutes)
-$activityPoller->doActivity(1440);
+$activityPoller->doActivity(1440);

src/activities/transcoders/BasicTranscoder.php

@@ -102,7 +102,9 @@ public function getAssetInfo($inputFilePath)
                 "Execution of command '".$ffprobeCmd."' failed.",
                 $this->activityLogKey
             );
-            return false;
+
+            throw new CpeSdk\CpeException("Unable to execute FFProbe to get information about '$inputFilePath'!",
+                self::EXEC_VALIDATE_FAILED);
         }
 
         if (empty($out)) {

src/utils/CommandExecuter.php

@@ -37,7 +37,7 @@ public function execute(
     {
         if ($logKey)
             $this->logKey = $logKey;
-        
+
         $this->cpeLogger->logOut("INFO", basename(__FILE__), "Executing: $cmd", $this->logKey);
 
         // Start execution of $cmd
@@ -95,7 +95,7 @@ public function execute(
                 echo ".";
                 flush();
             }
-            
+
             // Read prog output
             if (isset($pipes[1]) && $pipes[1]) {
                 $out = stream_get_contents($pipes[1], -1);
@@ -122,10 +122,12 @@ public function execute(
                                      basename(__FILE__),
                                      "Can't execute: $cmd. Exit Code: ".$procStatus['exitcode'],
                                      $this->logKey);
-            if ($allOut)
+            if ($allOut) {
                 $this->cpeLogger->logOut("ERROR",
                                          basename(__FILE__), "COMMAND STDOUT: ".$allOut,
                                          $this->logKey);
+                $allOut = null;
+            }
             if ($allOutErr)
                 $this->cpeLogger->logOut("ERROR",
                                          basename(__FILE__), "COMMAND STDERR: ".$allOutErr,