Sync queue index

[AndreaFrancis]

According to index usage, these indexes are not used very frecuently or can ve covered by another index:

• ("type", "dataset", "revision", "config", "split", "status", "priority"),
• ("priority", "status", "created_at", "namespace"),
• ("unicity_id", "-created_at", "status"), # 4440

I hid them to evaluate if the system will be affected once we delete them.

Adding these indexes in the code as they already exist in the server and are used by the queries:

• ("dataset", "status"),
• ("priority", "status", "created_at", "namespace", "difficulty", "unicity_id"),
• ("priority", "status", "created_at", "difficulty", "namespace"),
• ("unicity_id", "status", "-created_at"),
  See index usage report:

[
  {
    name: 'type_1_dataset_1_revision_1_config_1_split_1_status_1_priority_1',
    key: {
      type: 1,
      dataset: 1,
      revision: 1,
      config: 1,
      split: 1,
      status: 1,
      priority: 1
    },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("8"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: {
        type: 1,
        dataset: 1,
        revision: 1,
        config: 1,
        split: 1,
        status: 1,
        priority: 1
      },
      name: 'type_1_dataset_1_revision_1_config_1_split_1_status_1_priority_1',
      background: false
    }
  },
  {
    name: 'priority_1_status_1_created_at_1_difficulty_1_namespace_1',
    key: {
      priority: 1,
      status: 1,
      created_at: 1,
      difficulty: 1,
      namespace: 1
    },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("11302"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: {
        priority: 1,
        status: 1,
        created_at: 1,
        difficulty: 1,
        namespace: 1
      },
      name: 'priority_1_status_1_created_at_1_difficulty_1_namespace_1'
    }
  },
  {
    name: 'priority_1_status_1_created_at_1_namespace_1_difficulty_1_unicity_id_1',
    key: {
      priority: 1,
      status: 1,
      created_at: 1,
      namespace: 1,
      difficulty: 1,
      unicity_id: 1
    },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("275"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: {
        priority: 1,
        status: 1,
        created_at: 1,
        namespace: 1,
        difficulty: 1,
        unicity_id: 1
      },
      name: 'priority_1_status_1_created_at_1_namespace_1_difficulty_1_unicity_id_1'
    }
  },
  {
    name: 'priority_1_status_1_type_1_namespace_1_unicity_id_1_created_at_1_difficulty_-1',
    key: {
      priority: 1,
      status: 1,
      type: 1,
      namespace: 1,
      unicity_id: 1,
      created_at: 1,
      difficulty: -1
    },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("6030"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: {
        priority: 1,
        status: 1,
        type: 1,
        namespace: 1,
        unicity_id: 1,
        created_at: 1,
        difficulty: -1
      },
      name: 'priority_1_status_1_type_1_namespace_1_unicity_id_1_created_at_1_difficulty_-1'
    }
  },
  {
    name: '_id_',
    key: { _id: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("24007"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: { v: 2, key: { _id: 1 }, name: '_id_' }
  },
  {
    name: 'finished_at_1',
    key: { finished_at: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("0"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { finished_at: 1 },
      name: 'finished_at_1',
      background: false,
      expireAfterSeconds: 600,
      partialFilterExpression: { status: { '$in': [ 'success', 'error', 'cancelled' ] } }
    }
  },
  {
    name: 'status_1_type_1',
    key: { status: 1, type: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("59169"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { status: 1, type: 1 },
      name: 'status_1_type_1',
      background: false
    }
  },
  {
    name: 'type_1_dataset_1_status_1',
    key: { type: 1, dataset: 1, status: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("3252"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { type: 1, dataset: 1, status: 1 },
      name: 'type_1_dataset_1_status_1',
      background: false
    }
  },
  {
    name: 'priority_1_status_1_created_at_1_namespace_1',
    key: { priority: 1, status: 1, created_at: 1, namespace: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("10901"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { priority: 1, status: 1, created_at: 1, namespace: 1 },
      name: 'priority_1_status_1_created_at_1_namespace_1'
    }
  },
  {
    name: 'unicity_id_1_created_at_-1_status_1',
    key: { unicity_id: 1, created_at: -1, status: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("4440"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { unicity_id: 1, created_at: -1, status: 1 },
      name: 'unicity_id_1_created_at_-1_status_1'
    }
  },
  {
    name: 'dataset_1_status_1',
    key: { dataset: 1, status: 1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("406"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { dataset: 1, status: 1 },
      name: 'dataset_1_status_1'
    }
  },
  {
    name: 'unicity_id_1_status_1_created_at_-1',
    key: { unicity_id: 1, status: 1, created_at: -1 },
    host: 'atlas-x5jgb3-shard-00-01.ujrd0.mongodb.net:27017',
    accesses: { ops: Long("12700"), since: ISODate("2023-11-22T19:20:27.967Z") },
    spec: {
      v: 2,
      key: { unicity_id: 1, status: 1, created_at: -1 },
      name: 'unicity_id_1_status_1_created_at_-1'
    }
  }

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8dfe935) 90.20% compared to head (6afaa0b) 90.50%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2140      +/-   ##
==========================================
+ Coverage   90.20%   90.50%   +0.29%     
==========================================
  Files         265      250      -15     
  Lines       16216    15441     -775     
==========================================
- Hits        14628    13975     -653     
+ Misses       1588     1466     -122     

Flag	Coverage Δ
jobs_cache_maintenance	95.33% <ø> (ø)
jobs_mongodb_migration	86.69% <ø> (ø)
libs_libapi	?
libs_libcommon	92.03% <100.00%> (+0.40%)	⬆️
services_admin	86.56% <ø> (ø)
services_api	83.44% <ø> (ø)
services_rows	82.72% <ø> (ø)
services_search	80.48% <ø> (ø)
services_sse-api	94.21% <ø> (ø)
services_worker	91.73% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[severo]

Perfect.

Before merging, please delete the rows instead of commenting, we won't lose anything since it's versioned.