Renate Schaaf

I have worked on a port of my Bitmaps2Video-encoder to using Windows Media Foundation instead of ffmpeg, since I wanted to get rid of having to use all those dll's. Now, before posting it on GitHub, I'd like to run it by the community because of my limited testing possibilies. I also hope that there are people out there having more experience with MF and could give some suggestions on the problems remaining (see below). Learning how to use media foundation certainly almost drove me nuts several times, because of the poor quality of the documentation and the lack of examples. What is does: Encodes a series of bitmaps to video with the user interface only requiring basic knowledge about videos. Can do 2 kinds of transitions between bitmaps as an example of how to add more. Supports file formats .mp4 with encoders H264 or H265, or .wmv with encoder WMV3. Does hardware encoding, if your GPU supports it, falls back to software encoding otherwise. Uses parallel routines wherever that makes sense. Experimental routine to mux in mp3-audio. Only works for H264 and WMV3 right now. Requirements: VCL-based. Needs the excellent MF headers available at https://github.com/FactoryXCode/MfPack. Add the src-folder of MFPack to the library path, no need to install a package. Needs to run on Windows10 or higher to make use of all features. Not sure about Delphi-version required, guess XE3 and up is required for sure. Problems remaining: I'm not too thrilled about the encoding quality. Might be a problem with my nVidia-card. The audio-muxer should work for H265, because it works when I use ffmpeg. But with my present routine the result just plays the audio and shows no video. I haven't yet figured out how to insert video clips. Major problem I see is adjusting the frame rate. Renate Bitmaps2VideoWMF.zip

I have made my parallel resampling of bitmaps now as fast as I can get it. Now I would find it interesting to know, how the algorithm performs on other systems, and it would be super to get suggestions for improvement. The procedures can be found in the unit uScale under Algorithms in the attached zip. I have tested against Windows StretchBlt-Half_Tone, WICImage and Graphics32. On my PC (AMD Ryzen 7 1700, 8-core) I see a substantial improvement in speed. The threads are based on TThread rather than TTask or TParallel, because I had failures using the latter two, whereas the oldfashioned threads haven't failed me ever in gazillions of runs. If you want to compile and run the comparison project, you need to include the source folder of Graphics32 (latest version) in your search path. For convenience it is included in the zip under Algorithms. I couldn't find a way to divvy out only the units needed for resampling. The test against Graphics32 might be slightly unfair, because of bitmaps being assigned to TBitmap32 before the processing. Right now, the procedure itself cannot be run in concurrent threads, because the array of TThreads is a global variable, I need to change the design (ideas welcome). There might be still room for improvement by minimizing the cache misses, but my head can't handle it at the moment. Hope you find it useful. Renate Bitmap Scaling.zip

This is my video-project on GitHub: https://github.com/rmesch/Bitmaps2Video I am presenting it here, because it is useful as it is, but could use some ideas for improvement. Features: A Delphi-class to support encoding of a series of bitmaps and video clips to a video file Requires the ffmpeg-library and is intended as an easy to use interface to this library Versions for Win32/Win64 and a cross-platform version currently supporting Win32/Win64/Android32/Android64 Most popular file formats and codecs supported, more codecs contained in FFMpeg can be registered Rudimentary support for adding an audio-stream Demos for both versions, set up to compile and run "out of the box", as library files and support for their deployment to Android are included There are some problem areas though, the most important one in my opinion being threading issues with TBitmap under Android. For more see the readme and the demos. Critique, ideas, bug reports most welcome, maybe someone would even like to contribute, that would be delightful. There have been valuable contributions so far, but there are some areas which could use the input of an expert. Thanks for reading, Renate

I have update the repo on GitHub https://github.com/rmesch/Parallel-Bitmap-Resampler Changes made to the "modern" VCL- and FMX-version in the folder BitmapScaling: New resampling filters: Mitchell, Robidoux, RobidouxSharp, RobidouxSoft. Simplified and corrected MakeGaussContributors in uScaleCommon.pas. @Anders Melander: It will pass the uniform color tests now. But it will fail the Gauss-RMS, since I changed to RadiusToSigma back. Tried to make Gamma-correction a little more precise. I tried nonetheless. You already spent so much time digging through that ancient attachment, give the repo a shot. I also added the option in DemoScale.dpr to use a test-bitmap similar to yours. I can't see any of the color-artefacts you describe, though.

Right. You want to add 1 frame of your animation at a time, but you use bme.addStillImage, which is meant for adding the same image for multiple frames. So it will only work (roughly) correctly if the ShowTime is much larger than the frame time of the movie. Try to use bme.AddFrame instead. That just won't work, it's a codec limitation. You have to use at least even numbers, for some codecs the sizes might even have to be multiples of 4. I would stick to multiples of 4 to be on the safe side. Another thing you might consider is to shorten the chain from animation to movie. To show the animation and make screenshots seems a bit roundabout to me, there must be a shorter way. There must be, but I haven't yet bothered to look at it 🙂, maybe I will.

OK, Maple computed the following simplified filters, to implement them was just a matter of extending the TFilter-Enum. I'll update my repo some time tomorrow, the new filters need to be implemented in the demos. Right now I feel more like surviving a few more days on The Long Dark. // The following filters are based on the Mitchell-Netravali filters with // restricting the parameters B and C to the "good" line B + 2*C = 1. // We have eliminated B this way and scaled the filter to [-1,1]. // See https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_filters const C_M = 1 / 3; // Mitchell filter used by ImageMagick function Mitchell(x: double): double; inline; begin x := abs(x); if x < 0.5 then Result := (8 + 32 * C_M) * x * x * x - (8 + 24 * C_M) * x * x + 4 / 3 + 4 / 3 * C_M else if x < 1 then Result := -(8 / 3 + 32 / 3 * C_M) * x * x * x + (8 + 24 * C_M) * x * x - (8 + 16 * C_M) * x + 8 / 3 + 8 / 3 * C_M else Result := 0; end; const C_R = 0.3109; // Robidoux filter function Robidoux(x: double): double; inline; begin x := abs(x); if x < 0.5 then Result := (8 + 32 * C_R) * x * x * x - (8 + 24 * C_R) * x * x + 4 / 3 + 4 / 3 * C_R else if x < 1 then Result := -(8 / 3 + 32 / 3 * C_R) * x * x * x + (8 + 24 * C_R) * x * x - (8 + 16 * C_R) * x + 8 / 3 + 8 / 3 * C_R else Result := 0; end; .... and so on. Just one function with different constants.

I know! Just missed the B and C-values for the Robidoux in the table-image you post. And then I just have to rescale the functions to have support in [-1,1], make sure it's integral is 1. Bang. It plugs right in. Wish I could edit my original post and delete the attachment, it's ancient now, and include a link to my GitHub-repo. The AntiNLanczos is a spline to approximate the antiderivative of Lanczos, all that stuff isn't needed anymore.

They are in System.Math: function Min(const A, B: Integer): Integer; overload; inline; I had coded it with ifs in a previous version, but I changed that after I noticed the inlining, looks a bit less stupid. Oh, W is the weight you compute, and param is the x of the kernel. So you *did* post the kernel code, I was just too dense to see it. I think Maple and me can take it from there. I tried to find something in their source code, but gave up. Looks like you had a bit more stamina :).

What's the function these parameters need to be plugged into? All I can gather is that it might be some kind of cubic spline, and I don't feel like reading all of this guy's papers :). Would you mind posting the formula for the kernel?

MadExcept can check for frozen main thread, with or without debugger. Just surprised nobody mentioned it.

It doesn't only depend to the pitch, but also on the pixel-format of the source. If that is BGR or BGRA, the following pseudo-code based on what you post should be usable. If the color-order is different, like RGB or RGBA, you need to copy the single color-bytes. Best done by defining a record for the pixel. // Pointers to Pixels in Source/Target to be copied var pPixS, pPixT: pByte; // Bytes per Pixel for the Source-Texture // would be 3 for BGR, 4 for BGRA // if the channels have a different order, like RGB, // then the single color bytes need to be copied. var BytesPerPixel: integer; // Common Width and Height of Source and Target var Width, Height: integer; for I := 0 to Height - 1 do begin pPixS := @FTexture.pData[FTexture.RowPitch * I]; pPixT := FBitmap.Scanline[I]; for j := 0 to Width - 1 do begin Move(pPixS^, pPixT^, BytesPerPixel); inc(pPixS, BytesPerPixel); inc(pPixT, 4); end; end;

It's probably not. Your blur is slightly slower than mine for small radii. For large radii, yours is much faster. I'll mail you my benchmark-unit, then you can see for yourself.

Thanks very much for the input, I hadn't looked at those filters more closely before, should be easy to test them out. Thank you! Here is a first result for radius 10. It only passed the Gauss-RMS-test after I changed the sigma-to-radius-ratio to the same as yours. Need to give that a closer look. For other radii my routine failed some of the uniform color tests, (and edge detection as a consequence,) so it's back to the drawing board for that.

I managed to get it, source or not. For the same amount of "blurriness" my parallel version needs about 1.5 times the time of yours. Source would still be nice, I'm sure we'd learn something. Renate

I already did. I meanwhile learned how to do pull requests 🙂 Looking forward to the mailman! Renate

It won't work, unless the images are all of the same size. Changing the size would cause at least a partial re-initialization.

I was fooled by the names in the example-project. I just had a closer look and, yes it's box blur. As far as I can see GaussianBlur isn't Gaussian either, it just introduces weights of 1/square_distance_from_center. They do handle the alpha-channel, though. I would have liked to see the alpha-blended result, but I couldn't get the ImageView to show it. Seeing as my (unthreaded) routine is truly Gaussian, and that it outperforms GR32.GaussianBlur by a factor of roughly 2-3, would it make sense for me to suggest it as a replacement in GR32? How can I better test the quality? Where can I find your routine then? Quality usually takes precedence over performance for me, unless the performance annoys me, well it's also fun to get something faster 🙂

OK, I plugged my unsharp-mask into the Blurs-example of GR32. Doing so, made me aware of the need to do gamma-correction when you mix colors. So I implemented that, but see below. Also, I finally included options to properly handle the alpha-channel for the sharpen/blur. The repo at GitHub has been updated with these changes. Results: Quality: My results seem a tad brighter, otherwise I could see no difference between Gaussian and Unsharp. Performance: Unthreaded routine: For radii up to 8 Unsharp is on par with FastGaussian, after that FastGaussian is the clear winner. Threaded routine: Always fastest. If anybody is interested, I am attaching the test project. It of course requires to have GR32 installed. It also requires 10.3 or higher, I guess. Gamma-correction: I did it via an 8bit-Table same as GR32. This seems very unprecise to me, but I wouldn't know how to get it any more precise other than operating with floats, no thanks. Sadly, this can produce visible banding in some images, no matter which blur is used. Here is an example (for uploading all images have been compressed, but the effect is about the same): Original, a cutout from a picture taken with my digital camera. Result of Gaussian with Radius = 40 and Gamma = 1.6 When gamma-correction is used for sharpening, bright edge-artifacts are reduced, but dark edge-artifacts are enhanced. My conclusion right now would be to not use gamma-correction. But if anybody has an idea for how to implement it better, I'm all ears. Thanks, Renate BlurTest.zip

But you would have to tell the encoder that every frame is a key-frame, otherwise it only stores the differences. But it can be done ... Also, the access to the frames for decoding needs to be sped up. But you could this way create a stream of custom-compressed images. Of course, no other app would be able to use this format.

HEIF is a container format like .mp4, as far as I see Windows manages this file format via WICImage only. MFPack contains headers for this, but all that goes a bit over my head. If you want to test HEVC-compression, you can do this via BitmapsToVideoWMF by creating an .mp4-file with just one frame using the procedures below. This is anything but fast, because of the initialization/finalization of Mediafoundation taking a long time. A quick test compresses a .jpg of 2.5 MB taken with my digital camera to an .mp4 of 430 KB. No quality loss visible at first glance. uses VCL.Graphics, uTools, uTransformer, uBitmaps2VideoWMF; procedure EncodeImageToHEVC(const InputFilename, OutputFileName: string); var wic: TWicImage; bm: TBitmap; bme: TBitmapEncoderWMF; begin Assert(ExtractFileExt(OutputFileName) = '.mp4'); wic := TWicImage.Create; try bm := TBitmap.Create; try wic.LoadFromFile(InputFilename); WicToBmp(wic, bm); bme := TBitmapEncoderWMF.Create; try // Make an .mp4 with one frame. // Framerate 1/50 would display it for 50sec bme.Initialize(OutputFileName, bm.Width, bm.Height, 100, 1 / 50, ciH265); bme.AddFrame(bm, false); finally bme.Free; end; finally bm.Free end; finally wic.Free; end; end; procedure DecodeHEVCToBmp(const mp4File: string; const Bmp: TBitmap); var vi: TVideoInfo; begin vi := uTransformer.GetVideoInfo(mp4File); GetFrameBitmap(mp4File, Bmp, vi.VideoHeight, 1); end;

This is why that makes no sense (quoted from https://cloudinary.com/guides/video-formats/h-264-video-encoding-how-it-works-benefits-and-9-best-practices) "H.264 uses inter-frame compression, which compares information between multiple frames to find similarities, reducing the amount of data needed to be stored or transmitted. Predictive coding uses information from previous frames to predict the content of future frames, further reducing the amount of data required. These and other advanced techniques enable H.264 to deliver high-quality video at low bit rates. "

Do you mean to Jpeg or Png? A video-format would not make any sense for a single image. NVidea-chips can apparently do that, but I don't know of any Windows-API which supports that. If you want to encode to Png or Jpeg a bit faster than with TPngImage or TJpegImage use a TWicImage. Look at this nice post on Stackoveflow for a way to set the compression quality for Jpeg using TWicImage: https://stackoverflow.com/questions/42225924/twicimage-how-to-set-jpeg-compression-quality

I'll see what I can do. Stream might be tricky, I would need to learn more about this.

New version at https://github.com/rmesch/Parallel-Bitmap-Resampler: Has more efficient code for the unsharp-mask, and I added more comments in code to explain what I'm doing. Procedures with explaining comments: uScaleCommon.Gauss uScaleCommon.MakeGaussContributors uScaleCommon.ProcessRowUnsharp and see type TUnsharpParameters in uScale.pas. Would it be a good idea to overload the UnsharpMask procedure to take sigma instead of radius as a parameter? Might be easier for comparison to other implementations.

Hi Anders, It's great that you think of it, but hold off on that for a bit. I noticed that I compute the weights in a horrendously stupid way. The weights are mostly identical, it's not like when you resample, dumb me. So taking care of that reduces memory usage by a lot and the subsequent application of the weights becomes much faster. I've also changed the sigma-to-radius ratio a bit according to your suggestion. I find it hard to make results look nice with cutoff at half the max-value, I changed it to 10^-2 times max-value. But this still allows for smaller radii, and it becomes again a bit faster. So, before you do anything I would like to finish these changes, and also comment the code a bit more. (Forces me to really understand what I'm doing 🙂