How does Azure Service Bus identify a duplicate message?

The duplicate detection is looking at the MessageId property of the brokered message. So, if you set the message Id to something that should be unique per message coming in the duplicate detection can catch it. As far as I know only the message Id is used for detection. The contents of the message are NOT looked at, so if you have two messages sent that have the same actual content, but have different message IDs they will not be detected as duplicate.

References:

MSDN Documentation: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions

If the scenario cannot tolerate duplicate processing, then additionallogic is required in the application to detect duplicates which can beachieved based upon the MessageId property of the message which willremain constant across delivery attempts. This is known as ExactlyOnce processing.

There is also a Brokered Message Duplication Detection code sample on WindowsAzure.com that should be exactly what you are looking for as far as proving it out.

I also quickly tested this out and sent in 5 messages to a queue with RequiresDuplicateDetection set to true, all with the exact same content but different MessageIds. I then retrieved all five messages. I then did the reverse where I had matching MessageIds but different payloads, and only one message was retrieved.

In my case I have to apply ScheduledEnqueueTimeUtc on top of MessageId.Because most of the time the first message already got pickup by worker, before the sub-sequence duplicate message were arrive in the Queue.By adding ScheduledEnqueueTimeUtc. We tell the Service bus to hold on the the message for some time before letting worker them up.

            var message = new BrokeredMessage(json)            {                MessageId = GetMessageId(input, extra)            };            // Delay 30 seconds for Message to process            // So that Duplication Detection Engine has enought time to reject duplicated message            message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(30);

Another important property to be considered while dealing with 'RequiresDuplicateDetection' property of a Azure Service Bus entity is 'DuplicateDetectionHistoryTimeWindow', the time frame within which message with duplicate message id will be rejected.

Default value of duplicate detection time history now is 30 seconds, the value can range between 20 seconds and 7 days.

Enabling duplicate detection helps keep track of the application-controlled MessageId of all messages sent into a queue or topic during a specified time window. If any new message is sent carrying a MessageId that has already been logged during the time window, the message is reported as accepted (the send operation succeeds), but the newly sent message is instantly ignored and dropped. No other parts of the message other than the MessageId are considered.