Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #4922

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

Change feed: Adds id and pk to ChangeFeedMetadata for delete operations #4922

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
814d5b2
Update ChangeFeedItem to include id and pk metadata
jcocchi Dec 4, 2024
c8b47fc
update emulator tests for avad delete operations
jcocchi Dec 5, 2024
c9900c6
add output of UpdateContracts script
jcocchi Dec 11, 2024
b66a6a7
undo encryption contracts changes
jcocchi Dec 12, 2024
e38c25c
update changefeed metadata serialization
jcocchi Dec 12, 2024
673e86c
update ttl delete test
jcocchi Dec 12, 2024
1da602c
update change feed metadata contract
jcocchi Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Update ChangeFeedItem to include id and pk metadata
  • Loading branch information
@jcocchi
jcocchi committed Dec 4, 2024
commit 814d5b2bc7471a512e585aa5d568311a1e1c5f9e
6 changes: 3 additions & 3 deletions Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedItem.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,21 +57,21 @@ namespace Microsoft.Azure.Cosmos
class ChangeFeedItem<T>
{
/// <summary>
/// The full fidelity change feed current item.
/// The current version of the item for all versions and deletes change feed mode.
/// </summary>
[JsonProperty(PropertyName = "current")]
[JsonPropertyName("current")]
public T Current { get; set; }

/// <summary>
/// The full fidelity change feed metadata.
/// The item metadata for all versions and deletes change feed mode.
/// </summary>
[JsonProperty(PropertyName = "metadata", NullValueHandling = NullValueHandling.Ignore)]
[JsonPropertyName("metadata")]
public ChangeFeedMetadata Metadata { get; set; }

/// <summary>
/// For delete operations, previous image is always going to be provided. The previous image on replace operations is not going to be exposed by default and requires account-level or container-level opt-in.
/// The previous version of the item for all versions and deletes change feed mode. The previous version on delete and replace operations is not exposed by default and requires container-level opt-in. Refer to https://aka.ms/cosmosdb-change-feed-deletes for more information.
/// </summary>
[JsonProperty(PropertyName = "previous", NullValueHandling = NullValueHandling.Ignore)]
[JsonPropertyName("previous")]
Expand Down
17 changes: 15 additions & 2 deletions Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadata.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
namespace Microsoft.Azure.Cosmos
{
using System;
using System.Collections.Generic;
using System.Text.Json;
using Microsoft.Azure.Cosmos.Resource.FullFidelity;
using Microsoft.Azure.Cosmos.Resource.FullFidelity.Converters;
Expand All @@ -21,7 +22,7 @@ namespace Microsoft.Azure.Cosmos
#else
internal
#endif
class ChangeFeedMetadata
class ChangeFeedMetadata
{
/// <summary>
/// The change's conflict resolution timestamp.
Expand Down Expand Up @@ -50,9 +51,21 @@ class ChangeFeedMetadata
public long PreviousLsn { get; internal set; }

/// <summary>
/// Used to distinquish explicit deletes (e.g. via DeleteItem) from deletes caused by TTL expiration (a collection may define time-to-live policy for documents).
/// Used to distinguish explicit deletes (e.g. via DeleteItem) from deletes caused by TTL expiration (a collection may define time-to-live policy for documents).
/// </summary>
[JsonProperty(PropertyName = ChangeFeedMetadataFields.TimeToLiveExpired, NullValueHandling = NullValueHandling.Ignore)]
public bool IsTimeToLiveExpired { get; internal set; }

/// <summary>
/// The id of the previous item version. Used for delete operations only.
/// </summary>
[JsonProperty(PropertyName = ChangeFeedMetadataFields.DeletedItemId, NullValueHandling = NullValueHandling.Ignore)]
public string DeletedItemId { get; internal set; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that the backend only returns Deleted Item information for now but in the future it will also return update information and so on. Do you think it would be better to name this field as ItemId instead of DeletedItemId? We can update our documentation to reflect this fact.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Id and partition key in the metadata section will always only be for delete. I added the DeletedItem prefix to disambiguate that, otherwise customers may expect it to be populated for every operation.

You may be thinking of Previous which will eventually be populated for both deletes and updates.


/// <summary>
/// The partition key of the previous item version. Dictionary Key is the partition key property name and Dictionary Value is the partition key property value. Used for delete operations only.
/// </summary>
[JsonProperty(PropertyName = ChangeFeedMetadataFields.DeletedItemPartitionKey, NullValueHandling = NullValueHandling.Ignore)]
public Dictionary<string, string> DeletedItemPartitionKey { get; internal set; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend checking with Kiran/Fabien on how the Hierarchical keys are represented with in our codebase so that we are consistent but otherwise I would mark it as Dictionary<string, object> as the value could of any type.

}
}
2 changes: 2 additions & 0 deletions Microsoft.Azure.Cosmos/src/Resource/FullFidelity/ChangeFeedMetadataFields.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,7 @@ internal class ChangeFeedMetadataFields
public const string OperationType = "operationType";
public const string PreviousImageLSN = "previousImageLSN";
public const string TimeToLiveExpired = "timeToLiveExpired";
public const string DeletedItemId = "id";
public const string DeletedItemPartitionKey = "partitionKey";
}
}
22 changes: 22 additions & 0 deletions Microsoft.Azure.Cosmos/src/Resource/FullFidelity/Converters/ChangeFeedMetadataConverter.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
namespace Microsoft.Azure.Cosmos.Resource.FullFidelity.Converters
{
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Text.Json;
using System.Text.Json.Serialization;
Expand Down Expand Up @@ -56,6 +57,19 @@ public override ChangeFeedMetadata Read(ref Utf8JsonReader reader, Type typeToCo
{
metadata.PreviousLsn = property.Value.GetInt64();
}
else if (property.NameEquals(ChangeFeedMetadataFields.DeletedItemId))
{
metadata.DeletedItemId = property.Value.GetString();
}
else if (property.NameEquals(ChangeFeedMetadataFields.DeletedItemPartitionKey))
{
Dictionary<string, string> partitionKey = new Dictionary<string, string>();
foreach (JsonProperty pk in property.Value.EnumerateObject())
{
partitionKey.Add(pk.Name, pk.Value.GetString());
}
metadata.DeletedItemPartitionKey = partitionKey;
}
}

return metadata;
Expand All @@ -75,6 +89,14 @@ public override void Write(Utf8JsonWriter writer, ChangeFeedMetadata value, Json
writer.WriteNumber(ChangeFeedMetadataFields.Lsn, value.Lsn);
writer.WriteString(ChangeFeedMetadataFields.OperationType, value.OperationType.ToString());
writer.WriteNumber(ChangeFeedMetadataFields.PreviousImageLSN, value.PreviousLsn);
writer.WriteString(ChangeFeedMetadataFields.DeletedItemId, value.DeletedItemId);

writer.WriteStartObject("partitionKey");
foreach (KeyValuePair<string, string> kvp in value.DeletedItemPartitionKey)
{
writer.WriteString(kvp.Key, kvp.Value);
}
writer.WriteEndObject();

writer.WriteEndObject();
}
Expand Down
Loading