Bitmaps2Video for Windows Media Foundation

[Renate Schaaf 64]

I have worked on a port of my Bitmaps2Video-encoder to using Windows Media Foundation instead of ffmpeg, since I wanted to get rid of having to use all those dll's.
Now, before posting it on GitHub, I'd like to run it by the community because of my limited testing possibilies.

I also hope that there are people out there having more experience with MF and could give some suggestions on the problems remaining (see below).
Learning how to use media foundation certainly almost drove me nuts several times, because of the poor quality of the documentation and the lack of examples.

 

What is does:
  Encodes a series of bitmaps to video with the user interface only requiring basic knowledge about videos.
  Can do 2 kinds of transitions between bitmaps as an example of how to add more.
  Supports file formats .mp4 with encoders H264 or H265, or .wmv with encoder WMV3.
  Does hardware encoding, if your GPU supports it, falls back to software encoding otherwise.
  Uses parallel routines wherever that makes sense.
  Experimental routine to mux in mp3-audio. Only works for H264 and WMV3 right now.

 

Requirements:
  VCL-based.
  Needs the excellent MF headers available at https://github.com/FactoryXCode/MfPack.
    Add the src-folder of MFPack to the library path, no need to install a package.
  Needs to run on Windows10 or higher to make use of all features.
  Not sure about Delphi-version required, guess XE3 and up is required for sure.

 

Problems remaining:
  I'm not too thrilled about the encoding quality. Might be a problem with my nVidia-card.
  The audio-muxer should work for H265, because it works when I use ffmpeg. But with my present routine the result just plays the audio and shows no video.
  I haven't yet figured out how to insert video clips. Major problem I see is adjusting the frame rate.

 

Renate

Bitmaps2VideoWMF.zip

[Anders Melander 1783]

1 hour ago, Renate Schaaf said:

Needs the excellent MF headers available at https://github.com/FactoryXCode/MfPack.
    Add the src-folder of MFPack to the library path, no need to install a package.

I would suggest you add it as a Git submodule.

[Renate Schaaf 64]

1 hour ago, Anders Melander said:

I would suggest you add it as a Git submodule.

Thanks, sounds like it's just the thing needed. Just need to figure out how.

 

Error: uToolsWMF uses a unit Z_prof. Entry can be safely deleted. Comes from having too much stuff in the path.

 

Meanwhile I think I figured out how to improve the encoding quality.

 

Around line 270 in uBitmaps2VideoWMF.pas make the following changes:

 

if succeeded(hr) then
    hr := MFCreateAttributes(attribs, 4);  //<--------- change to 4 here
  // this enables hardware encoding, if the GPU supports it
  if succeeded(hr) then
    hr := attribs.SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, UInt32(True));
  // this seems to improve the quality of H264 and H265-encodings:
  {*************** add this *********************************}
  // this enables the encoder to use quality based settings
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonRateControlMode, 3);
  {**************** /add this *******************************}
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonQuality, 100);
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonQualityVsSpeed, 100);

Besides, I'm getting sick and tired of all those if succeeded...
 

[programmerdelphi2k 237]

@Renate Schaaf

 

is it not possible just this way?  or pSample.XXXXX( XXXXX ) can break next line = exception, for example?

  if (MFCreateSample(pSample) = S_OK) then // S_OK = 0
    begin
      hrSampleBuffer   := pSample.AddBuffer(pSampleBuffer);
      hrSampleTime     := pSample.SetSampleTime(fWriteStart);
      hrSampleDuration := pSample.SetSampleDuration(fSampleDuration);
      hrWriteSample    := pSinkWriter.WriteSample(fstreamIndex, pSample);
      //
      if (hrSampleBuffer or hrSampleTime or hrSampleDuration or hrWriteSample) = S_OK then // or just -> (hrWriteSample = S_OK) then
        begin
          inc(fWriteStart, fSampleDuration);
          fVideoTime := fWriteStart div 10000;
          inc(fFrameCount);
        end
      else
        raise Exception.Create('TBitmapEncoderWMF.WriteOneFrame failed');
    end;

 

Edited May 27, 2023 by programmerdelphi2k

[Renate Schaaf 64]

@programmerdelphi2k

What if hrSampleBuffer is not S_Ok, causing pSinkWriter.WriteSample to fail with an AV or such? You would get an exception the source of which would be harder to trace.

 

Or am I just too dense to understand :). (It's late)

 

 

 

[programmerdelphi2k 237]

I was just asking you if that way would work for you, you know? 

5 hours ago, Renate Schaaf said:

Besides, I'm getting sick and tired of all those if succeeded...

 

2 hours ago, Renate Schaaf said:

causing pSinkWriter.WriteSample to fail with an AV or such?

well, "MFCreateSample(pSample)" = "Initially the sample does not contain any media buffers".

then after this line, it will. "SetSampleTime" and "SetSampleDuration" just define the time values, "WriteSample(fstreamIndex, pSample)", just set the index of new Sample, then, I think that it will works, because the "pSample" already filled above by MFCreateSample()! and if  "NOT (MFCreateSample(pSample) = S_OK) then nothing will be done below... or you can raise a exception if you desire...

  if (MFCreateSample(pSample) = S_OK) then // S_OK = 0
  //...
  else
    raise Exception.Create('TBitmapEncoderWMF.WriteOneFrame failed');

 

if (hrSampleBuffer or hrSampleTime or hrSampleDuration or hrWriteSample) = S_OK = if (0 or 0 or 0 or 0) = 0 ... any thing non 0 = false

I have downloaded you sample and I did the changes above and it worked. 

 

https://learn.microsoft.com/en-us/windows/win32/api/mfapi/nf-mfapi-mfcreatesample

https://learn.microsoft.com/en-us/windows/win32/api/mfobjects/nf-mfobjects-imfsample-setsampletime

https://learn.microsoft.com/en-us/windows/win32/api/mfobjects/nf-mfobjects-imfsample-setsampleduration

https://learn.microsoft.com/en-us/windows/win32/api/mfobjects/nf-mfobjects-imfsample-addbuffer

https://learn.microsoft.com/en-us/windows/win32/api/mfreadwrite/nf-mfreadwrite-imfsinkwriter-writesample

Edited May 27, 2023 by programmerdelphi2k

[Renate Schaaf 64]

You are right. But then I have to think about whether or not a call is likely to produce havoc, and declare several new variables.

BTW: WriteSample pushes the sample to the sinkwriter and does not just set an index.

What do you think about the following construct instead:

 

procedure TBitmapEncoderWMF.WriteOneFrame;
var
  pSample: IMFSample;
  Count: integer;

const
  ProcName = 'TBitmapEncoderWMF.WriteOneFrame';
  procedure CheckFail(hr: HResult);
  begin
    inc(Count);
    if not succeeded(hr) then
      raise Exception.Create('Fail in call nr. ' + IntToStr(Count) + ' of ' +
        ProcName + ' with result ' + IntToStr(hr));
  end;

begin
  Count := 0;
  // Create a media sample and add the buffer to the sample.
  CheckFail(MFCreateSample(pSample));

  CheckFail(pSample.AddBuffer(pSampleBuffer));

  CheckFail(pSample.SetSampleTime(fWriteStart));

  CheckFail(pSample.SetSampleDuration(fSampleDuration));
  // Send the sample to the Sink Writer.
  CheckFail(pSinkWriter.WriteSample(fstreamIndex, pSample));

  inc(fWriteStart, fSampleDuration);
  fVideoTime := fWriteStart div 10000;
  inc(fFrameCount);
end;

 

Edited May 28, 2023 by Renate Schaaf

[programmerdelphi2k 237]

8 hours ago, Renate Schaaf said:

What do you think about the following construct instead:

I think that is not problem too... the big question "would be"?

• where the "exception" (using the pSample in any function above "addBuffer, SetSampleTime, SetSampleDuration, WriteSample") would generated, exactly?
  • my question is about that in MS links dont show if an "exception" would be generated (exactly), but just say the "result would be <> 0", then, if I has the "pSample" = ok, what happens if "I cannot set the Time, Duration" for example, but I can "write a new sample without this 2 values?" for example. The new sample would be invalid or it will use default values? if not invalid then write it would be ok not?
• using your "CheckFail()" I think that is the same, because you are executing "CheckFail" after "CheckFail", then, if the 1 fail, what happens to others? you see? the same than my "vars = xxxxxx"....
• now, if none "exception" is "generated by this functions ("addBuffer, SetSampleTime, SetSampleDuration, WriteSample") then, you choice what is better for you, because your "CheckFail" only raise the exception AFTER "HR=FALSE", or be, you raise the exception, not the "function" itself, you see?

[maXcomX 3]

The sample only works on Delphi 10.3 and above.

[maXcomX 3]

5 hours ago, programmerdelphi2k said:

I think that is not problem too... the big question "would be"?

• where the "exception" (using the pSample in any function above "addBuffer, SetSampleTime, SetSampleDuration, WriteSample") would generated, exactly?
  • my question is about that in MS links dont show if an "exception" would be generated (exactly), but just say the "result would be <> 0", then, if I has the "pSample" = ok, what happens if "I cannot set the Time, Duration" for example, but I can "write a new sample without this 2 values?" for example. The new sample would be invalid or it will use default values? if not invalid then write it would be ok not?
• using your "CheckFail()" I think that is the same, because you are executing "CheckFail" after "CheckFail", then, if the 1 fail, what happens to others? you see? the same than my "vars = xxxxxx"....
• now, if none "exception" is "generated by this functions ("addBuffer, SetSampleTime, SetSampleDuration, WriteSample") then, you choice what is better for you, because your "CheckFail" only raise the exception AFTER "HR=FALSE", or be, you raise the exception, not the "function" itself, you see?

Checking the results of the media foundation methods is as easy as can be. The results will give you information about what went wrong in most cases.  So, the standard exception handling in Delphi is not suitable and therefore you have to write our own exception handlers, because each hresult code is described in WinApi.WinError.pas or WinApi.MediaFoundationApi.MfError.pas and will exactly return what went wrong. Media Foundation and related API's are not part of the Delphi distributions from Embarcadero, and if they are, they are outdated in most cases. The Microsoft documentation explains the retuned values of the API methods very well.

[maXcomX 3]

On 5/27/2023 at 8:39 PM, Renate Schaaf said:

Thanks, sounds like it's just the thing needed. Just need to figure out how.

 

Error: uToolsWMF uses a unit Z_prof. Entry can be safely deleted. Comes from having too much stuff in the path.

 

Meanwhile I think I figured out how to improve the encoding quality.

 

Around line 270 in uBitmaps2VideoWMF.pas make the following changes:

 

if succeeded(hr) then
    hr := MFCreateAttributes(attribs, 4);  //<--------- change to 4 here
  // this enables hardware encoding, if the GPU supports it
  if succeeded(hr) then
    hr := attribs.SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, UInt32(True));
  // this seems to improve the quality of H264 and H265-encodings:
  {*************** add this *********************************}
  // this enables the encoder to use quality based settings
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonRateControlMode, 3);
  {**************** /add this *******************************}
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonQuality, 100);
  if succeeded(hr) then
    hr := attribs.SetUINT32(CODECAPI_AVEncCommonQualityVsSpeed, 100);

Besides, I'm getting sick and tired of all those if succeeded...
 

🙂 Learn C++ I would say.  Delphi users are quite spoiled about resolving method results. But helaas, until now, there is no solution in Delphi to handle HResult in it's exception handlers for media foundation and directx. To be more specific: WinApi.Error.Pas is way outdated until version 10.3

[maXcomX 3]

On 5/27/2023 at 10:51 PM, programmerdelphi2k said:

@Renate Schaaf

 

is it not possible just this way?  or pSample.XXXXX( XXXXX ) can break next line = exception, for example?

  if (MFCreateSample(pSample) = S_OK) then // S_OK = 0
    begin
      hrSampleBuffer   := pSample.AddBuffer(pSampleBuffer);
      hrSampleTime     := pSample.SetSampleTime(fWriteStart);
      hrSampleDuration := pSample.SetSampleDuration(fSampleDuration);
      hrWriteSample    := pSinkWriter.WriteSample(fstreamIndex, pSample);
      //
      if (hrSampleBuffer or hrSampleTime or hrSampleDuration or hrWriteSample) = S_OK then // or just -> (hrWriteSample = S_OK) then
        begin
          inc(fWriteStart, fSampleDuration);
          fVideoTime := fWriteStart div 10000;
          inc(fFrameCount);
        end
      else
        raise Exception.Create('TBitmapEncoderWMF.WriteOneFrame failed');
    end;

 

And will this give any user or debugger information about what exactly went wrong? The HResult code does. So, you have to be more specific when raising an exception, I would think.

[programmerdelphi2k 237]

said so much, and nothing!

[Renate Schaaf 64]

1 hour ago, maXcomX said:

Learn C++ I would say. 

No thanks 🙂. But I've already translated parts of C++-code to Delphi. They could use stuff like Break_On_Fail(hr) and more, made me a bit jealous.

 

You wouldn't per chance know why I can't mux any audio into an HEVC-encoded video? The video stream is all there, but it seems to be missing the correct stream header. So only the audio is being played.

[maXcomX 3]

1 hour ago, Renate Schaaf said:

No thanks 🙂. But I've already translated parts of C++-code to Delphi. They could use stuff like Break_On_Fail(hr) and more, made me a bit jealous.

 

You wouldn't per chance know why I can't mux any audio into an HEVC-encoded video? The video stream is all there, but it seems to be missing the correct stream header. So only the audio is being played.

Maybe you forgot to add a stream for sound? Container formats always have separate streams for sound and/or subtitles. That's why most players can play a video in native languages and subtitles.

Edited May 29, 2023 by maXcomX
Clarifying

[maXcomX 3]

Quote

They could use stuff like Break_On_Fail(hr) and more, made me a bit jealous.

In C++ they are called macro's. A thing Delphi does not have.

[maXcomX 3]

On 5/27/2023 at 11:47 PM, Renate Schaaf said:

@programmerdelphi2k

What if hrSampleBuffer is not S_Ok, causing pSinkWriter.WriteSample to fail with an AV or such? You would get an exception the source of which would be harder to trace.

 

Or am I just too dense to understand :). (It's late)

 

 

 

In that case assign the buffer to nil (which is silence) and try again. However an av will never happen unless the API is wrong translated or when you use sloppy code. You will always get an HResult when something goes wrong.

[Renate Schaaf 64]

10 hours ago, maXcomX said:

However an av will never happen unless the API is wrong translated or when you use sloppy code.

Well, I must have written lots of sloppy code yesterday, but I can now encode audio to AAC together with video to H264 or H265. I'm using the CheckFail approach wherever possible. I wish I could keep all these attribute names in my head. Wonder whether it would be possible collecting them into records, so you could consult code completion about them.

[maXcomX 3]

On 5/27/2023 at 11:47 PM, Renate Schaaf said:

6 hours ago, Renate Schaaf said:

Wonder whether it would be possible collecting them into records, so you could consult code completion about them.

I use a special app that reads almost all possible result codes and when possible where to find more details. These codes and descriptions are stored in Delphi units.

 

 

 

[Renate Schaaf 64]

A first version of TBitmapEncoderWMF is now available at

 

https://github.com/rmesch/Bitmaps2Video-for-Media-Foundation

 

It only supports .mp4 and codecs H264 and HEVC(H265) at the moment. See the readme for more details.

 

I had some problems getting the video-stream to use the correct timing. Apparently the encoder needs to have the video- and audio-samples fed to it in just the right way,

otherwise it drops frames or changes timestamps at its own unfathomable judgement.

I think I solved it by artificially slowing down the procedure that encodes the same frame repeatedly, and by reading ahead in the audio-file for "just the right amount".

 

Would be interested in how it fares on other systems than mine.

[Kas Ob. 121]

On 8/15/2023 at 10:36 AM, Renate Schaaf said:

I had some problems getting the video-stream to use the correct timing. Apparently the encoder needs to have the video- and audio-samples fed to it in just the right way,

otherwise it drops frames or changes timestamps at its own unfathomable judgement.

Some problems !, one might call WMF the mother of all problems that can encode and decode.

I remember trying use the encoder in real time for live video chat, and dropped it later and never looked back.

 

Anyway, i can't compile and run TBitmapEncoderWMF, but i looked at the source and found this the following line

 

 

This to my experience with the whole Windows Media Foundation that approach is a problem, and most likely a core problem in your encoder.

The problem in such calculation with that your are not using audio sample rate and the ability to divide the audio sample right, so my suggestion is to to round that fSampleDuration to what audio sample allow you and will be acceptable to the encoder and of course to the decoder in later stage.

 

To understand the problem and forgive my English and inconsistent phrases, you have to know why the standard of the audio sample rate for many many years was 44100 hz, why this number and from where it came, please read this https://en.wikipedia.org/wiki/44%2C100_Hz

 

and see its there for compatibility with PAL at 50 fps and NTSC at 60 fps, so to make it right in your encoder you should choose fSampleDuration according to acceptable/right audio sample rate, that will add to exact (real exact like 44100) audio sample rate per second.

 

Also you should make sure that audio sample rate and the sample rate buffer have the right channels, you can't have audio sample rate with 2 channels and feed the encoder an odd number of samples !, same goes for 5.1 channels or 7.1, just food for thoughts, if and most likely you are doing 2 channels (Strereo) then the audio sample per frame should always be even number.

 

By the way, it is beautiful code what you did there !

[Renate Schaaf 64]

4 hours ago, Kas Ob. said:

The problem in such calculation with that your are not using audio sample rate and the ability to divide the audio sample right, so my suggestion is to to round that fSampleDuration to what audio sample allow you and will be acceptable to the encoder and of course to the decoder in later stage.

I think I know what you mean, I'll play around with it some. Problem is that I really need to have a well defined video-frame-rate, otherwise the user can't time the input. I have meanwhile figured out the audio-sample-duration for given sample-rate and use it for the amount of audio-samples to be read ahead. Audio-sync is now acceptable to me. Problem is now, that some decoders are sloppyly implemented in Windows, like .mpg.

 

4 hours ago, Kas Ob. said:

the audio sample per frame should always be even number.

I'm not sure I even have control over that. Isn't the encoder just using everything that is in the leaky bucket to write the segments in what it thinks is the right way?

 

Why can't you compile the code? I think your feedback would be very valuable to me. Did you try the latest version on GitHub?

https://github.com/rmesch/Bitmaps2Video-for-Media-Foundation

(there's one silly bit of code still in there that I need to take out, but it usually doesn't hurt anything)

I was hoping to have eliminated some code that prevented it to run on versions earlier than 11.3. Or have you meanwhile developed an aversion against the MF-headers? 🙂

Edited September 5, 2023 by Renate Schaaf
Wrong link

[Kas Ob. 121]

58 minutes ago, Renate Schaaf said:

Why can't you compile the code?

The last Delphi IDE I have is Seattle, and these are unknown to both of us (me and the IDE)

 

1 hour ago, Renate Schaaf said:

Or have you meanwhile developed an aversion against the MF-headers? 🙂

No, it was just researching for live video long time ago, and was completely sure that WMF is worthless as it, CPU usage was unacceptable high for ASF video while the quality was good, also synching the audio with video was big problem as i needed to use my own jitter to fast play the audio when a delay in traffic over wire were introduced, the result was no for it.

The whole API give you the sense it was built for specific Microsoft software, unlike Windows libraries.

 

1 hour ago, Renate Schaaf said:

I think your feedback would be very valuable to me.

Well i don't think you need help you are doing great job, please feel free to share any question to brain storm it together.

[Renate Schaaf 64]

The use of TVirtualImageList is totally unessential, just cosmetics. I have replaced it by the old TImageList now. Hope it works, and thanks!

[chmichael 12]

Can you add:

1) Encode to Stream

2) Hardware encode only 1 image ?

Thank you