Issue with CrowdStrike Connector : PRECONDITION_FAILED Error, Epoch Timestamps Cause Errors

[avinashKumarYadav]

Part 1: Bug/Issue

Description:
The CrowdStrike connector in OpenCTI encounters the following issues when using fixed epoch timestamps for data ingestion:

1. Indicators Not Pulled:
   Setting CROWDSTRIKE_INDICATOR_START_TIMESTAMP to any value other than 0 (e.g., for pulling indicators for the last 6 hours) results in no indicators being ingested.

2. Error in Logs:
   The connector logs display the following error:

   PRECONDITION_FAILED - message size 53319771 is larger than configured max size 16777216
   

3. Mismatched Timestamps for Reports/Actors:
   Reports and actors are pulled, but their timestamps do not match the defined epoch values and differ significantly from what is configured.

Environment:

• OS: debian 12 (bookworm)
• OpenCTI Version: 6.4.5
• Setup: Dockerized (All containers)

Reproducible Steps:

1. Configure fixed epoch timestamps for indicators, reports, and actors (e.g., last 6 hours for indicators, last 7 days for reports/actors).

• CROWDSTRIKE_INDICATOR_START_TIMESTAMP=1735042879
• CROWDSTRIKE_ACTOR_START_TIMESTAMP=1734459679
• CROWDSTRIKE_REPORT_START_TIMESTAMP=1734459679

2. Start the connector.
3. Observe:
   • No indicators are ingested.
   • The PRECONDITION_FAILED error appears in the logs.
   • Reports and actors are pulled but with timestamps outside the specified epoch range.

Expected Output:

• Indicators, reports, and actors are ingested as per the configured epoch timestamps without errors.

Actual Output:

• Indicators fail to ingest.
• The connector logs show PRECONDITION_FAILED.
• Reports and actors are ingested, but their timestamps differ from the configured values.

---

Part 2: Suggestion/Feedback and Query

Use Case:
Our operational requirement is to:

• Pull indicators for only the last 6 hours.
• Pull reports and actors for only the last 7 days.

Currently, the connector relies on epoch timestamps (e.g., CROWDSTRIKE_INDICATOR_START_TIMESTAMP), but setting these timestamps manually is:

1. Prone to the issues/errors mentioned above.
2. Time-consuming and not practical for recurring dynamic requirements like pulling data for the "last 6 hours" or "last 7 days."

Query and Suggestions:

1. How can we configure the CrowdStrike connector to dynamically pull data for specific relative time periods (e.g., last 6 hours, last 7 days) instead of relying on fixed epoch timestamps?
2. Are there any built-in features or recommended methods to support dynamic configurations for such use cases?
3. If this is not currently supported, we suggest implementing a feature allowing users to define relative time ranges directly in the configuration (e.g., last 6 hours, last 7 days) rather than setting fixed epoch timestamps.

---

This configuration is critical for our operations. Kindly provide guidance on addressing these issues and implementing the desired use case. Let us know if additional details are required.

[avinashKumarYadav]

Clarifying the environment details to avoid any potential confusion due to a typo in the earlier message:

OS: Debian 12 (Bookworm)
OpenCTI Version: 6.4.5
Setup: Dockerized (All containers)