[BUG] Setting fragment data not working

[daveisfera]

Media pipeline:

• What is the media source (GStreamer, custom, etc). What is the upstream encoder/pipeline? Samples

Logging
I added this to the sample so you can see everything by running directly:
#394

Describe the bug
Setting the fragment data returns that it's working, but it doesn't appear in the fragments. It's occasionally present on the first fragment, but never present on any of the following fragments.

SDK version number
1.5.0

Open source building
N/A

To Reproduce
(Use the updated sample found here: #394)

1. Set fragment metadata
2. Send video frame
3. Observe errors

Expected behavior
Metadata is present in all of the fragments where it's been added

Desktop (please complete the following information):

• OS: Debian Bookworm
• Browser [e.g. chrome, safari]: N/A
• Version [e.g. 22]: N/A

Additional context
Sorry that I didn't explain this more directly in #396 but even though the errors are no longer present, the functionality still isn't working

[jdelapla]

In your sample what is an example input for each parameter so I can reproduce this?

[daveisfera]

I made it so the options have defaults when not specified, so you can use to turn on the fragment metadata and still run with the default setup:

./kvsVideoOnlyRealtimeStreamingSample <stream_name> "" "" "" 10

[jdelapla]

So I've checked out your fork and run the sample. I also exported the debug data dump, here is a snippet of the MKV:

+ EBML head
|+ EBML version: 1
|+ EBML read version: 1
|+ Maximum EBML ID length: 4
|+ Maximum EBML size length: 8
|+ Document type: matroska
|+ Document type version: 2
|+ Document type read version: 2
+ Segment: size unknown
|+ Segment information
| + Segment UID: 0xa8 0x1c 0xeb 0xf0 0x2f 0x58 0xcb 0x00 0xc2 0x2b 0x34 0xd3 0x53 0x7a 0xa7 0x7a
| + Timestamp scale: 1000000
| + Title: Kinesis Video SDK
| + Multiplexing application: Kinesis Video SDK 1.1.0
| + Writing application: Kinesis Video SDK 1.1.0
|+ Tracks
| + Track
|  + Track number: 1 (track ID for mkvmerge & mkvextract: 0)
|  + Track UID: 1
|  + Track type: video
|  + Name: kvs_video_track
|  + Codec ID: V_MPEG4/ISO/AVC
|  + Video track
|   + Pixel width: 640
|   + Pixel height: 480
|  + Codec's private data: size 45 (H.264 profile: High @L3.0)
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_0
|   + String: 0
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_1
|   + String: 119388
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_2
|   + String: 2
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_3
|   + String: 119390
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_4
|   + String: 4
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_5
|   + String: 119392
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_6
|   + String: 6
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_7
|   + String: 119394
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_8
|   + String: 8
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_9
|   + String: 119396
|+ Cluster
| + Cluster timestamp: 470561:31:36.386000000
| + Cluster position: 0
| + Simple block: key, track number 1, 1 frame(s), timestamp 470561:31:36.386000000
|  + Frame with size 119389
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.426000000
|  + Frame with size 15776
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.466000000
|  + Frame with size 15221
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.506000000
|  + Frame with size 15644
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.546000000
|  + Frame with size 18064
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.586000000
|  + Frame with size 14735
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.626000000
|  + Frame with size 16516
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.666000000
|  + Frame with size 13544
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.706000000
|  + Frame with size 13612
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.746000000
|  + Frame with size 14018
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.786000000
|  + Frame with size 13752
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.826000000
|  + Frame with size 13759
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.866000000
|  + Frame with size 14309
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.906000000
|  + Frame with size 14923
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.946000000
|  + Frame with size 13788
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:36.986000000
|  + Frame with size 16577
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.026000000
|  + Frame with size 17412
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.066000000
|  + Frame with size 17425
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.106000000
|  + Frame with size 17451
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.146000000
|  + Frame with size 19960
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.186000000
|  + Frame with size 20493
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.226000000
|  + Frame with size 21082
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.266000000
|  + Frame with size 21524
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.306000000
|  + Frame with size 21733
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.346000000
|  + Frame with size 21563
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.386000000
|  + Frame with size 21327
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.426000000
|  + Frame with size 21278
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.466000000
|  + Frame with size 19842
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.506000000
|  + Frame with size 18849
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.546000000
|  + Frame with size 17988
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.586000000
|  + Frame with size 18287
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.626000000
|  + Frame with size 18514
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.666000000
|  + Frame with size 18282
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.706000000
|  + Frame with size 19958
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.746000000
|  + Frame with size 20465
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.786000000
|  + Frame with size 20144
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.826000000
|  + Frame with size 20194
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.866000000
|  + Frame with size 20614
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.906000000
|  + Frame with size 20467
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.946000000
|  + Frame with size 20360
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:37.986000000
|  + Frame with size 20075
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:38.026000000
|  + Frame with size 19171
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:38.066000000
|  + Frame with size 19576
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:38.106000000
|  + Frame with size 18769
| + Simple block: track number 1, 1 frame(s), timestamp 470561:31:38.146000000
|  + Frame with size 20105
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_0
|   + String: 45
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_1
|   + String: 103999
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_2
|   + String: 47
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_3
|   + String: 104001
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_4
|   + String: 49
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_5
|   + String: 104003
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_6
|   + String: 51
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_7
|   + String: 104005
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_8
|   + String: 53
|+ Tags
| + Tag
|  + Simple
|   + Name: TEST_KEY_9
|   + String: 104007
|+ Cluster

You can see that all 10 metadata items are appended, and the pattern continues for every cluster. So there isn't an issue present. The KVS service has a limit of 10 unique metadata key/value items per fragment https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/how-meta.html , and there was an issue with metadata previously being pre-pended and post-pended to the cluster based off timing of the metadata submission, which could have caused your test sample to post more than 10 fragments to a single cluster and give the error discussed in the previous ticket.

The stated error of this ticket is that every fragment should have metadata and from multiple runs of your provided test code that appears to be the case. Resolving ticket.

[daveisfera]

Where are you looking at that MKV data and how are you viewing it? When I load the fragments with GET_MEDIA and look at them with ffprobe, the tags are not present, so are looking at the values in the KVS Producer SDK before they're sent to KVS or are you pulling them back out and verifying that they were correctly stored and available in the data?

[sharadrj]

@daveisfera , we are prioritizing this issue for investigation this week. Apologies for not being able to pick it up earlier.

[hassanctech]

@daveisfera I checked with ffprobe as well and I do not see the metadata, however if you check with mkvinfo -v flag you will see it. With out the -v flag it will not show all the information. Can you please try mkvinfo -v on what you retrieved from get media. I tried it and I'm able to see it. If it's not there can you please upload an example with (1) the file that's dumped by the SDK and (2) what you downloaded when calling get media.

[daveisfera]

I can verify that in some cases mkvinfo will show tags that ffprobe doesn't, but they're not always there like they're supposed to be and the AWS tags always show up, like AWS_KINESISVIDEO_FRAGMENT_NUMBER so shouldn't the tags added by the metadata always show up in the same way?

[hassanctech]

I was unable to reproduce this, I ran a cycle of adding 1 tag, then 2 tags then 3 tags, ..., 10 tags, and they were always there. One thing I did see is once the 11th tag is added, the tags 1-10 get removed (when setting the persistent flag to TRUE). I did not try with the persistent flag set to FALSE. Also make sure you use the -v flag with mkvinfo otherwise you will only get tags that appear before the first cluster if it happened to be there. To see the rest of the tags you need the -v option and you should see all of them.

[daveisfera]

The sample that I submitted to reproduce the problem uses the persistent flag set to both TRUE and FALSE and neither of them persist. I just re-ran that test and the first fragment has the tags but the rest don't. I verified that with both ffprobe and mkvinfo -v

[hassanctech]

If we have 3 things that would be really helpful:

1. export the env var KVS_DEBUG_DUMP_DATA_FILE_DIR locally, this will result in a MKV dump from the SDK from the curl thread that's actually sending the frame data, and please provide us that file
2. Full SDK VERBOSE level logs during this time (which will include all the fragment ACKS, which includes fragment timecodes and a fragment number)
3. Call this API: https://docs.aws.amazon.com/cli/latest/reference/kinesis-video-archived-media/get-media-for-fragment-list.html to pull down the MKV fragment. If you can provide the MKV you get from here as well that would be great.

This way we can see the MKV that was actually sent to the server, the MKV that is returned from the server and the SDK side logs which tell us the fragment ack information as well as if the fragment metadata related APIs failed (make sure in your application you are logging the result of the put fragment metadata call so we know it is actually succeeding).

[daveisfera]

I've been using GetMedia and when I use GetMediaForFragmentList is returns the following rather than the fragment data (I tried with both aws-cli 2.13.21 and boto3 1.28.59 and both give this response):

{"Output":{"__type":"com.amazon.coral.service#UnknownOperationException"},"Version":"1.0"}

Is this a known issue or am I doing something wrong?

[sirknightj]

Hi @daveisfera, could you please share the aws cli command you're using? (with any credentials redacted)

[daveisfera]

aws kinesis-video-archived-media get-media-for-fragment-list test.mp4 --stream-name dljtest --fragments "91343852333181783967653219705270632698727947677"

[unicornss]

Would you please try sending the get-media-for-fragment-list with the endpoint-url ?

Example:

aws kinesis-video-archived-media  get-media-for-fragment-list  --stream-name your-test-test --fragments='[ "913438523...21158223834954403","913438523331821504....79790303014943","91343852333....68492004201388098766007","91343852333....128649145722987760357888","913438523331821....06287244593940623010"]'  output-kvs.mkv --endpoint-url  "https://b-aaa-ccc.kinesisvideo.us-west-2.amazonaws.com"

aws cli reference:
https://docs.aws.amazon.com/cli/latest/reference/kinesis-video-archived-media/get-media-for-fragment-list.html

Note
You must first call the GetDataEndpoint API to get an endpoint. Then send the GetMediaForFragmentList requests to this endpoint using the --endpoint-url parameter .

or as pointed out by @sirknightj when using boto3

session = boto3.Session()
client = session.client('kinesis-video-archived-media', endpoint_url=x)

[daveisfera]

I got the boto3 version get_media_for_fragment_list to work and I can confirm that the metadata tags are present, but they're attached to the end of the previous fragment and not the beginning of the current fragment. For the first fragment, they are in the right place, but after that they're in the "wrong place"
That's why ffprobe doesn't find them, so can this be fixed so they're always in the right place and properly detectable by standard tools?

[unicornss]

Could you share the results of the mkvinfo -v -v -v output.mkv command (https://mkvtoolnix.download/doc/mkvinfo.html) for a single specific fragment?

[daveisfera]

I attached the output from the first and second fragment. You can see that the first has two sets of tags (one at the beginning and then one at the end) and then the second fragment only has the set of tags at the end
first.txt
second.txt

[daveisfera]

And then for completeness, here's the last one and you can see that it doesn't have any tags because there were two sets in the first fragment
last.txt