separate Laravel log channels/Cloudwatch log groups

[rufhausen]

Is it not possible to push log events to separate Cloudwatch log groups?
Our application currently a number of Laravel Log channels for different 3rd party integrations, etc., and we push those to their own log groups in Cloudwatch using the AWSLogs service running on the app EC2s.
In the past we tried using the AWS PHP SDK to create a custom Cloudwatch Laravel log driver to push logs directly to cloudwatch, but it couldn't keep up with the logs we were generating.

[mnapoli]

I wouldn't recommend that indeed because that would send the logs over the HTTP API (I think), which is much slower. Not sure how to achieve what you are trying to do here to be honest.

[rufhausen]

So far, my solution is to add some context to the single log group that reflects the specific channel in logging.php being used. Starting with the need to make $logger->name public. With this, I can update the filters in our logging destinations DataDog/Cloudwatch to break out the entries based on the existing log channels in the app.

use Monolog\Logger as MonologLogger;

class Logger extends MonologLogger
{
    public $name;
}

        'marketscan-log' => [
            'driver'  => 'custom',
            'channel' => 'marketscan',
            'via'     => ServerlessLogger::class,
        ],

class ServerlessLogger
{
    public function __invoke(array $config)
    {
        $channel       = $config['channel'] ?? 'default';
        $logger        = new Logger($channel);
        $handler       = new StreamHandler('php://stderr');
        $jsonFormatter = new JsonFormatter();
        $jsonFormatter->setMaxNormalizeDepth(20);
        $handler->setFormatter($jsonFormatter);
        $logger->pushProcessor(new ContextProcessor($logger));
        $logger->pushHandler($handler);

        return $logger;
    }
}

class ContextProcessor implements ProcessorInterface
{
    private $logger;

    /**
     * @param \Monolog\Logger $logger
     */
    public function __construct(Logger $logger)
    {
        $this->logger = $logger;
    }

    /**
     * @param array $record
     *
     * @return array|null
     */
    public function __invoke(array $record): ? array
    {
        $record['context']['channel']  = $this->logger->name;
        $record['context']['env']      = config('app.env');
        $record['context']['hostname'] = (string) gethostname();

        return $record;
    }
}

\Log::channel('marketscan')->info('foo')

{"message":"foo","context":{"channel":"marketscan","env":"local","hostname":"c72d36bc94e6"},"level":200,"level_name":"INFO","channel":"local","datetime":"2022-06-30T11:23:48.009013-06:00","extra":{}}

[t-richard]

I think that's the way to go 👍

You can do quite powerful things with cloud watch logs insights or Datadog filtering to query those correctly.

Not related: Are you using Datadog php agent with Bref?

[rufhausen]

I wish I could have grabbed the name of the Laravel log channel instead of having to add a "channel" key/value, which is duplication, but couldn't find a way to access that in the Processor.

We haven't attempted to add the Datadog Agent yet. We're still figuring out if we can move to Bref, but's looking promising. Will looking adding the agent to Lambda.