MTFMapper results differ significantly from sfrmat
Status: Beta
Brought to you by:
fvdbergh
Menu
▾
▴
Hi Frans,
I’m trying out MTFMapper on a single ROI image crop (attached) and getting results that differ considerably from what I obtain with sfrmat.
The MTF estimates at Nyquist (90.91 lp/mm) are:
MTFMapper: ~30%
sfrmat5: ~19%
sfrmat3: ~21%
I also tried a couple of tools which gave different results again:
Mitre SFR 1.4.2: ~24%
ImageJ: ~10%
The target is approximately at best focus (~2m distant). There is noticeable FPN, and the target was just taped (not glued) onto a board, so you can see some variation in intensity due to that.
Do you have any idea of what’s going on?
Thank you,
-Larry
Hi Larry,
I took a quick look, and your SFR actually looks pretty good: a sharp lens that is focused well!
I think the root cause of the differences is the ICC Tone Response Curve (TRC) embedded in your PNG file. MTF Mapper identifies this as a 16-bit file with an ICC 'curv' type TRC, meaning the 16-bit integers in the PNG file are not a linear function of light intensity. Early on in the MTF Mapper processing pipeline it will use the ICC TRC to linearize the input data before performing any SFR analysis.
So the quick fix is that you can save your PNG with a linear profile to obtain more comparable results between MTF Mapper and the other tools. Not sure what software you used, but generally all you would have to do is change the colour space to a linear one (gamma=1.0).
Anyhow, I have attached two SFR plots extracted from your "center-crop.png" file (the alternate file appears to be corrupted?), one with default settings, and the other with the "linear gamma" option turned on in the Preferences dialog of MTF Mapper, forcing MTF Mapper to assume the input is linear so it will produce results more similar to sfrmat. Note that this is not the correct solution to the problem, I just include it as a quick example to show how much of a difference the ICC profile can make.
I have also attached two synthetic images generated with mtf_generate_rectangle (included with MTF Mapper), one with an sRGB ICC profile (roughly gamma=2.4), and another with a linear ICC profile (gamma = 1.0). You can process both these images with MTF Mapper and the other SFR tools to see what difference the non-linear TRC makes, since that's the only difference between the two images.
Thanks very much Frans,
I saw the option for linear gamma in the MTFMapper GUI, but from the check box text I assumed it only applied to 8 bit images.
The image actually comes from what is essentially an industrial camera, so the source image doesn't have any color profile information. It must get added somewhere down the line, possibly when I cropped it in Gimp. I've now tried saving it without a color profile, but PNG seems surprisingly stubborn in hanging onto gamma (which always seems to be 0.4545). I also tried ImageMagick, but again couldn't get rid of it. I also tried TIFF.
The only way I found to run without the "Linear gamma (8 bit)" option was to convert it to PGM, which obviously has no color profile or gamma metadata.
The MTF at Nyquist with linear gamma is virtually identical to what Mitre SFR produces, although Mitre's software only handles 8 bit images.
Any thoughts on how 8 bit vs 16 bit affects accuracy?
For whatever its worth, here are links to the various SFR tools I compared:
And I agree, the ImageJ plugin is suspect... it's buggy and I don't trust the results.
Btw, I'm not sure what the "alternate" file is, that wasn't intentional... just an artifact of my mail client.
-Larry
Ok, great!
I think using PGM is a decent workaround --- just keep in mind that MTF Mapper will assume that 8-bit images without any ICC profile are encoded with the sRGB TRC (which is usually the right choice), so that is why you still need the "linear gamma" flag. Conversely, 16-bit images without ICC profiles are assumed to be linear; this behaviour is mostly backwards compatibility with the original behaviour dating more than 10 years back now!
Another pitfall that I encountered recently is that some Basler camera models will apply a non-linear TRC to captured images even while the camera's LUT is disabled, and the Gamma is set to 1.0. Specifically, if you enable the "Light source preset" then there is an implicit hidden non-linear TRC. So while I like industrial camera modules, I now distrust them until I have proven that their "raw" data is in fact unprocessed. /rant :)
With regards to the the 16-bit vs 8-bit question: I just use 16-bit linear images whenever I can because there are some subtle differences in the results. Images captured with a real camera are probably dithered a little by noise, but on synthetic noise-free images I can demonstrate the differences easily --- I have attached three samples for you to play with, as well as a quick peek of the resulting SFR curves with some of the default MTF Mapper smoothing disabled (hint: the green curve was the 8-bit sRGB image, and the orange one was the 16-bit linear image).
Anyhow, I should probably change the Preferences dialog to clarify the effect of the "linear gamma" option.
Regards,
Frans
ps: splitting this post into two because the attachments were "spam", apparently
... and the rest of the attachments.
Hi there,
I was about to start a new thread but maybe this one applies to me.
I've just discovered MTF Mapper and I'm impressed with the job you did. Well done!
However, I was checking with ImageJ and Imatest Master and I find different values, so I'm not sure what's happening here...
I'm loading the attached image captured with a Nikon D610 (24Mpx) and I'm getting:
* 92.58 lp/mm in MTF Mapper (see attached capture)
* 41.8 C/P (which I guess is the same) in Imatest (see capture)
* About 0.41 in ImageJ (see capture)
So... why am I getting so high values with MTF Mapper?
If 92.58 line pairs is a true value, then with this camera+lens you can get up to 184 lines per mm. As the sensor measures 35.9mm width, then it could achieve 6605 vertical lines, which is impossible because with 24Mpx the image is 6016px width... so something doesn't match (if I haven't done wrong math).
I've read what you explained about ICC linearization. This file is in the sRGB space and it's icc tagged, so I guess all three are using the right information...
Any help would be nice. Thanks!
Hugo
I tried linearizing the image and the results are the same in MTF mapper, so it confirms what you explained, Frans
Glad to hear you have resolved the discrepancies! Feel free to ask more questions if you have any.
Regards,
Frans
Thanks for replying so fast, Frans.
However, I think I explained myself badly: what I really wanted to say is that after linearizing I got the same results that I already got, so the discrepancies with Imatest and ImageJ are still there.
This confirms that the software linearizes the images in the first stages, that was what I was referring for...
Regards,
Last edit: hugo rodriguez 2025-07-06
As you can see, with Imatest I get 41.8 lp/mm, about 39 with ImageJ and 92 with MTF Mapper. I thought it could be a simple matter of doubling numbers (lines instead of pairs) but there is still a difference...
Hi Hugo,
Apologies for the delay, I'm travelling so my responses may be a bit spotty.
I think we have to sort out the different units first. Imatest is reporting cycles per pixel, which are very different from the line pairs per mm you are reporting on via MTF Mapper.
In MTF Mapper you can just change the units in the settings (untick lp/mm units). For the lp/mm units to function correctly you also need to set the pixel pitch correctly (I think it should be 5.97 micron for the D610). But for comparing results to Imatest just unselect the lp/mm box for now so that MTF Mapper will report cycles per pixel.
Hopefully that will resolve the discrepancy.
Regards,
Frans
Hi Frans, thanks for your kind reply.
Ok, I guessed that cycles and line pairs give the same values to my understanding. In the end, a complete cycle goes to full white, then to full black and to the starting point... like a pair of lines. But I'll research a bit for more information, I could be wrong.
I still have a few questions if you don't mind:
1) Ok, so pixel pitch. To calculate it what data do I exactly need: the final effective pixels or photodetectors?
For the D610, with 6016 px width and 35.9mm it is: 5.96um but if we consider photodetectors it's 5.90um (6080 photodetectors).
2) Now testing again with the 5.96um factor it gives a result of 0.442 c/p which is very close to Imatest (0.418) but still not the same.
ImageJ gives almost the same number (0.41) but it clearly states that it's lp/mm...
So why this inconsistencies?
After inputting the pixel pitch, the lp/mm looks much more real, though (74 lp/mm)
Thanks a lot.
PS: After disabling the option for 'linear gamma' in preferences, I loaded some images and the program seemed to ignore it... until I restarted it again. Maybe a small bug?
Last edit: hugo rodriguez 2025-07-07
Hi Hugo,
I'll look into the potential bug, thanks for reporting it.
Regards,
Frans
Hi Frans, thanks a lot for your kind reply.
1) Well, I don't know the sensor model but I used RAWDigger to dig into the manufacturer metadata and there I found the value I was looking for: 6080 photo detectors. I assume it's right. Not a big difference, though, between 6016.
2) Regarding Imatest, I've just realized that when I entered 5.96um as the pixel pitch, the units were 'pixel per inch' instead of 'microns per pixel' which I think is the right one. See attached image.
However, this didn't made a big change in the measurements at all, except the Cy/mm (which went from 0.09835 cy/mm to 70.01 cy/mm). The main value (in bold) stay almost fixed at 0.418 C/P in both cases.
MTF Mapper gives 0.44 C/P which seems close now but still not the same.
I also did a test loading both gamm 2.2 and linear gamma images into Imatest and the results were quite different. I'm attaching the results (note: these captures show the wrong value of 5.97 px/inch that can be seen at the bottom of the capture, as I realized about my mistake a bit later).
Also in the corner, now both MTF Mapper and Imatest are quite close to each other, although not perfectly. And I still don't get what's the difference between lp/mm and C/mm...
I can send you the original image if you wish, for your own testing.
Best regards,
Hugo
PS: just a small suggestion: is it too complicated to make the Open... window remember its size when you reopen it? And what about seeing all your Disk units in the left panel?
one line pair is a black line and a white line; this can also be interpreted as one cycle, if you place two line pairs next to each other, then the distance from the middle of the first black line to the middle of the second black line is one cycle, however, this is the same distance as yhe width of a black line and a white line together. In other words, a line pair is the same thing as a cycle.
when we measure in line pairs per mm, we are talking about physical line pairs as the lens would project them onto the sensor. Note that this does not involve pixels at all, but mm on the sensor.
On the other hand we can easily imagine that a line pair is scaled to the width of a sensor pixel, e.g., 5.97 micron. If one line pair is the same width as a pixel, we have one line pair per pixel, which is also one cycle per pixel. This would of course be a higher spatial frequency than what the pixel can resolve, so normally we consider something like 0.5 c/p to be a pretty high spatial frequency, especially for MTF50.
Hope that clears it up!
Anyway, you can run my PNG example above, test_16bit_linear.png through both MTF Mapper and Imatest to compare their measurements on a clean slanted edge with a known MTF50 value (see earlier posts above). Like I mentioned, I am travelling, but I would be interested in a copy of your .NEF file (you can email it if you prefer) to look into this discrepancy myself when I get a chance.
Hi Frans,
That's right, and that's exactly what I always thoughit was: a cycle is the same as a pair of lines.
I also did a research in Imatest documentation and confirmed what I did know: a cycle is the same as a pair of lines.
Anyway, when you say that " If one line pair is the same width as a pixel, we have one line pair per pixel, which is also one cycle per pixel" I guess you mean a 'pair of pixels' instead of a single pixel because it's simply impossible to have a cycle per pixel.
Or maybe this is a language matter, that we in spanish say in other words.
I ran your 16bit png into Imatest and MTFM and I got these results. 0.549 in Imatest and .464 in MTFM. Had to state that this time I loaded the linear file 'as-is', without noticing Imatest that it was linear.
Then I repeated the test by telling Imatest that the gamma is 1 and got a different result: 0.3892
When loading the 8bit file I got the same as MTFM: 0.463
So it seems that is very important to tell Imatest what the encoded gamma of the file is to get proper results, while MTFM is much more robust in this field (which is essential, of course).
I attached screen captures of everything as you can see. No problem in sending you the file (it's not my photo so I only have the DNG, not the NEF file, but I can ask for it if you really need it.) Just tell me your email and I can send it to you.
Thanks a lot, Frans!
Hi Hugo!
Please send the file to fvdbergh@gmail.com, and I will take a look when my travels are over.
But to get back to the cycles/pixel issue, I really did mean one cycle fits into a pixel. If you had a perfect lens then the sensor will simply capture a 50% gray value ... but that is only for a very literal interpretation of 1.0 cycle per pixel. If you look at the slanted-edge method then the SFR is just the contrast as a function of spatial frequency, and we have now chosen to represent spatial frequencies in units of c/p. So if we want an image that has an MTF50 value of 1.0, all we need is an image that has 50% contrast (SFR = 0.5) at that frequency. If we were talking about a literal black-and-white line pattern, then we would be in trouble. But we only need an image of a step edge at, say, 5 degrees to apply the slanted-edge method. And since the slanted-edge method has a built-in oversampling method that can easily measure the SFR at frequencies above 1.0 c/p this is no problem at all.
In fact, you can generate an example image with something like this:
You can run this through MTF Mapper and you should get back an MTF50 of 1.0 c/p. If you examine the image closely you will notice that it appears to have very jagged edges, exactly what you would expect of an aliased image.
So 1.0 c/p is indeed higher than what the sensor can capture accurately, or without aliasing.
On the other hand, a 1-pixel black, 1-pixel white line pattern represents a frequency of 0.5 c/p (since the cycle length is 2 pixels). This is normally called the Nyquist frequency of the sensor, which represents the highest frequency the sensor can capture without aliasing (assuming a grayscale sensor) in the vertical or horizontal directions.
Hope that clears up some finicky details!
-Frans
Hi Frans!
Thanks for such a detailed response. Well, all that you explained is the same as I already knew, so I think we are talking the 'same language', great then :)
I couldn't create that image with that code as I'm no expert with command line. I wrote this into the program files/MTF mapper:
mtf_mapper.exe mtf_generate_rectangle -m 1.0 -n 0 -o sharp.pngBut got and error... any help will be welcomed :)
Regarding the differences between Imatest, sfrmat and MTFMapper I found an article of a friend, Jose Pereira, the author of ImageQA (https://github.com/jpereiranet/imageQA), which also uses the same algorithm of sfrmat and he analyzed the differences using synthetic slanted edge squares and real shots. That article opened my mind and made me think there is not an absolute reference on this matter.
Here's the article (a must reading, indeed)
http://imageqa.jpereira.net/ES/blog1_mtf_issues.php
So then after reading that, I'm only concerned about how do you do the conversion from Cycles/pixel to line pairs/mm.
Using my simple math it should be like:
C/P x (image width in px) / (sensor width in mm)
So with my test image (taken with a D610 and a Yongnuo 100mm, developed with ACR) it gives 0.442 (at center) and 74.15 lpmm in MTFM
Then it would be:
0.442 x 6016px / 35.9mm = 74.07
So then I think everything is fine, it seems correct.
So... why then ImageJ gives a value of lpmm similar to what MTFMapper gives in C/P? Good question...
As I know personally Carles Mitjà, main head of E_MTF_2xNyquist for ImageJ (he also lives here at Barcelona), I'll ask him directly :)
Hi again, Hugo!
Try
mtf_generate_rectangle.exe -m 1.0 -n 0 -o sharp.pngin the directory where mtf_mapper.exe lives.For converting between c/p and lp/mm, I use
c * (1000 / p)where c is the c/p value and p is the sensor pitch in micron, e.g., 0.442 * (1000 / 5.97) = 74.04 lp/mm, which is close enough to your value to show your formula is correct (exact dimensions of the sensor and number of pixels explain the remaining differences).There are a great number of details that affect the accuracy of MTF measurements, including:
1. Are pixel values in linear intensity units, or have they been converted correctly (the original topic here),
2. Is the edge orientation estimated correctly (many implementations already fail at this step, especially with noisy images)
3. Is the MTF calculation corrected for larger orientation angles (e.g., MITRE is not corrected); this specifically relates to edges oriented at larger angles in the sensor corners
4. Is the MTF calculation corrected for slightly curved edges, such as caused by lens distortion
and the list goes on. The synthetic images generated by MTF Mapper have been tested by other researchers, and have been found to have the correct SFRs to a high degree of accuracy (when used correctly, e.g. linear gamma), so if a slanted edge implementation does not measure the expected MTF50 value, or more generally the correct SFR curve, then it is more likely that the slanted edge implementation is incorrect (or is being used incorrectly).
I had a look at that blog post you shared on comparing SE implementations, and I am a little surprised by the differences. There could be a few contributing factors:
* An edge angle of 1 degree is very shallow; I would first do baseline testing at 5 degrees before trying 1 degree. In real world lens measurements I would always use edge orientations of at least 3 degrees
* By default the mtf_generate_rectangle tool will generate rougly gamma 2.2 8-bit PNG files, which will not be handled correctly by most other SE implementations. The blog post really should list the mtf_generate_rectangle command actually used in an appendix. So who knows the details, but I would recommend running mtf_generate_rectangle with the "-l" option to produce gamma 1.0 images for better compatibility with other SE implementations.
I'd be happy to review the details of such a comparison experiment if you ever have to design or run one.
-Frans
Ok, I'm sending you the files I used through WeTransfer for this test as well as some RAW from a Tamrom 16-30mm with a SONY a7R II at all apertures.
Please confirm me that you receive the email. It will expire in 3 days, but don't worry if you can't download it. I can also upload then to GDrive if you need.
All these is because I'm helping a friend, the man behind this website that specializes in lens testing: www.digitalcamaralens.com
Actually I'm comparing this particular lens:
https://www.digitalcamaralens.com/Html/Objetivos/Yongnuo/100_2.0/Yongnuo-100_2_Analisis.htm
He needs to optimize the workflow for testing because it takes so much time in boring tasks. I'm trying to convince him to use MTFM instead of ImageJ with SE_MTF_2xNyquist. But he is worried by seeing how different are the results specially in the corners, where CAs are present, as well as astigmatism and distortion.
So... how different can the results be (in the corner) when the slanted egde is not exactly at 8 or 10º because of the lens distortion (barrel or pincushion), vignetting or simply because the camera had a slight misalignment of less than 1º?
Regards,
Hmm. I don't want to go into too many details regarding SE_MTF since I have never looked at the code myself. But I do suspect that SE_MTF may not handle edge orientations larger than 5 degrees correctly; the earlier versions of the ISO 12233 standard simply required the edge to be at roughly 5 degrees, and many implementations just assume you stick to those parameters. It is entirely possible to use most other edge orientation angles with some minor changes to the algorithm. So that could explain some differences. I would say that rotating the camera to be within about 2 degrees of the nominal 5 degrees should be good enough to completely remove alignment errors as a source of potential error when comparing implementations (meaning the errors are so small they can be ignored)
You also mention distortion, which could have a large impact if no correction for curved edges is implemented. You can easily test this by restricting your analysis to a short edge section, e.g., 30 pixels as measured along the edge. If this short section gives the same results as the full edge, then distortion (curved slanted edge) is not a problem for that specific set-up, meaning lens, camera, target, distance, and sensor position. But I am pretty sure the older versions of SE_MTF would not handle curved edges. By the way, by curved edges I mean as little as 1 pixel deviation from a true straight line is enough to ruin the measurements.
MTF Mapper does not have a mechanism for dealing with vignetting. The impact of vignetting can be meaningful, but it has to result in a strong gradient over a distance of 64 pixels across the slanted edge. In practice I have only seen this a handful of times, but "slowly changing" vignetting is not usually a significant source of error.
The last topic, chromatic aberration, is a complicated one. Personally I treat this as a separate aberration, so I only use raw Bayer images, and select one Bayer channel at a time for MTF analysis. MTF Mapper will measure the MTF correctly in each channel under these conditions. But that is when I am evaluating a lens. If you are evaluating a whole system, including the raw image development software, then it may be valid to use MTF Mapper on a developed RGB image by setting the "Bayer channel" setting to "none". This will force MTF Mapper to convert the RGB image to Luminance and perform the measurements on the Luminance channel. If you have severe chromatic aberration in the lens corner you may start to see its impact of MTF values, which may be a valid thing to evaluate at a system level (leaving my purist approach aside for a moment).
If you truly want to compare SE implementations under these conditions it may be better to generate the Luminance image beforehand, either in your raw developer or using your tool of choice, and then evaluate the same image with all your SE implementations as a grayscale image.
I guess in the end there is little more I can say; obviously I believe that MTF Mapper is the free tool with built-in corrections for the largest number of potential sources of error (aside from vignetting); Next I would trust Imatest, and then sfrmat, in that order. When in doubt, use synthetic images with known properties to systematically test specific phenomena one at a time.
If you just use some actual camera captured images without ground truth, then there is no way to determine which software is giving you the correct results. You would have to perform your own set of experiments to systematically build up trust in your results. MTF Mapper has some history in this regard involving numerous independent researchers and optical engineers. That does not prove it will be correct under all possible test conditions,
but it does rule out most glaring errors.
But of course I would say this, so you have no choice but to do your own validation experiments 😀
Thanks for you kind, long and so well explained response, really. I'll do some testing!
Yes, MTF Mapper will directly process raw Bayer images. You can start with a raw camera file such as a .NEF or .CR2, and the MTFM GUI will call dcraw (or libraw) to render the image to a grayscale Bayer-mosaic Tiff file. This only happens when you set the Bayer channel option in the settings, and you'll see an icon next to the filename in the MTFM results list to indicate what type of channel interpretation was used (RGB, single channel grayscale, or Bayer red, green or blue).
If you are using the command-line version of MTFM, then you have to perform the camera raw file to Bayer mosaic Tiff conversion yourself.
But performing per-channel MTF analyses is somewhat specialized; I think most use-cases are fine with an RGB image since the Luminance channel is heavily weighted towards the green channel. If you want to investigate longitudinal chromatic aberration, though, the Bayer analysis is the way to go.
I'll be out for a few days, will reply you when I'm back :)