# Image stitching in Matlab

The image below is a panorama taken at the residence where I am living. It is made of 3 different photos which are stitched together using a simple Matlab program that can be downloaded from here, or github.

Image stitching: result

This program is, in fact, a simple programming exercise I have completed recently. The algorithm is quite straightforward:

1. The input images are fed into a Harris corner detector. In the code, the implementation of Harris detector does not do multi-scale detection, i.e. it is only a single-scale corner detector. It also does not detect interesting points at subpixel accuracy.
2. Based on detected interesting points, 128-dimensional SIFT descriptors are extracted. The implementation only gives the descriptors which are not rotation-invariant, i.e. it does not try to normalize the dominant orientation of the gradients.
3. For each pair of 2 images, the feature vectors are matched using the nearest neighborhood algorithm. The distance being used is the Euclidean distance (in 128-dimensional space), and the threshold for matching follows Lowe’s simple approach.
4. The matched feature vectors are then fed into RANSAC, which computes the homography transformation between two images, given the set of matched feature vectors.
5. The images are projected (using the estimated homography transformation) in order to get the final result.

The code includes several components which are not mine:

• harris.m: Harris corner detector by A. Efros
• find_sift.m: SIFT descriptor by S. Lazebnik
• and some others.

I only implemented matching and RANSAC, and some other fancy stuffs. Therefore I was able to complete it in several hours, yet the results are quite nice. While running, the program might display some intermediate results, like the set of tentative correspondences (pairs of matched points between two images) and the inliers (estimated in RANSAC). Inliners is always subset of tentative correspondences. A sample result is shown below.

Tentative correspondences and inliners

Using the script is easy:

```
> stitch({'cite/cite_a.jpg', 'cite/cite_b.jpg', 'cite/cite_c.jpg'}

```

The code currently supports 3 images, but extending that to multiple images is easy (by applying the chain rule for homography matrices).

Some other implementations that are available on the web:

Certainly, they are more beautifully implemented than mine, but I am still sharing the code, since it shows that we still can do something nice even with some minimal efforts.

Probably in the next posts, I might come back to some algorithms used in this program, especially RANSAC, a cute algorithm for fitting models.

Advertisements

## 56 comments

1. Tèo says:

Chào anh Vu, anh có tài liệu nào về Harris Corner Detector anh chia sẽ cho em với em đang làm về nó
Cảm ơn anh đã đọc tin

2. Tèo says:

cảm ơn anh Vu

3. Yao-Ling says:

Thanks for your awesome & clear direction about how to stitch multiple images together 🙂

By the way, I experience a issue while executing your source code:

Read cite/cite_a.jpg: ans =
600 800 3
Error using reshape
To RESHAPE the number of elements must not change.
Error in colfilt (line 181)
b(i*mb+brows,j*nb+bcols) = …
Error in harris (line 28)
Rmax = colfilt(R,[3 3],’sliding’,nonmax); % find neighbrhood max
Error in getSift (line 26)
[x, y, ~] = harris(img);
Error in stitch (line 6)
[sifts, siftLoc, images] = getSift(imgFiles);

May I have your kindly suggestion how to solve this problem? (My platform version of Matlab is R2013b). Please feel free to answer my inquire. Thank you very much:)

1. PHAM Hoai Vu says:

Hi Yao-Ling,

I have just tried the code on github, it works well. Please try again with this command:

>> stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

There is no “cite” photos on github, but I included “keble”, so you can give it a try. If it doesn’t work, just let me know.

Please note that there is a tweak on line 9 of warp.m, where I manually tune the position of the bounding box so that the images are projected correctly. You might want to change those values if you test the code on your own photos.

2. PHAM Hoai Vu says:

And I think the code on github already supports more than 3 photos, you can just give it a try.
I am not sure about the zip package on box.com, it is not well maintained.

1. Yao-Ling says:

Sorry for giving my testing result. Also, thanks for your prompt and kindly reply:)

I spent some times trying to find the possible solution. Yet, the result currently is still not working due to the same issue I face. (I adapted your suggestion – using the code on ‘github’ instead)

May I inquire which platform version of MATLAB that you use to process this programme? As I learned, ‘colfilt.m’ is embedded in the ‘Imagetool box’. Not sure whether this is the problem or not… if some parts of code are modified in MATLAB R2013b.

3. vidhya says:

i need how to top row is depth map and bottom row is error map for left to wright side output of image

4. mons says:

hello sir,
from which function i have start, harris .m is working then after which have to call ?
If i wish to call getsitch.m then should be mine input parameter?

please help me out…

Thanks and regards

5. mons says:

stitch({‘cite/cite_a.jpg’,’cite/cite_b.jpg’,’cite/cite_c.jpg’})

it giving me error :-

Error using stitch (line 17)
Not enough input arguments.

Error in m1 (line 1)
stitch({‘cite/cite_a.jpg’,’cite/cite_b.jpg’,’cite/cite_c.jpg’})

sir please help me out

6. mons says:

function [sifts, siftLocations, images] = getSift(imgFiles, maxWidth)

if i want to call this function , which input arguments should i have to pass??

7. mons says:

Sir,
While running your code which i found at
https://github.com/phvu/misc/tree/master/imageStitch

while the function:

stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

i am getting this :

>> m2
Read keble/keble_a.jpg:
ans =

568 720 3

Running find_sift
Read keble/keble_b.jpg:
ans =

568 720 3

Running find_sift
Read keble/keble_c.jpg:
ans =

568 720 3

Running find_sift
Out of memory. Type HELP MEMORY for your
options.

Error in dist2 (line 24)
n2 = (ones(ncentres, 1) * sum((x.^2)’,
1))’ + …

Error in match (line 6)
dist = dist2(imgFeatures1,
imgFeatures2);

Error in getHomography (line 14)
matches = match(imgFeatures1,
imgFeatures2, MATCH_THRESHOLD);

Error in computeHomoInfo (line 21)
[H, corres_1, corres_2,
inlierIdx] = …

Error in stitch (line 6)
homoInfo =
computeHomoInfo(info.features,
info.featureLocations, info.images);

Error in m2 (line 1)
stitch({‘keble/keble_a.jpg’,
‘keble/keble_b.jpg’,
‘keble/keble_c.jpg’})

1. PHAM Hoai Vu says:

It simply means what it says: You don’t have enough memory for the program to execute. Please consider running the code on a machine with more RAM.
In case your machine have enough RAM, then you might need to increase the memory limit of MATLAB.

8. mons says:

Thank you sir for reply

9. sunkan says:

Hi im having a few problem with my image results.

i have 3 png images they are already grayed out.

when i run the program i get the following

stitch({‘hotel-00.png’,’hotel-01.png’,’hotel-02.png’})
Read hotel-00.png:
ans =

600 800

Error using rgb2gray>parse_inputs (line 81)
MAP must be a m x 3 array.

Error in rgb2gray (line 35)
X = parse_inputs(varargin{:});

Error in harris (line 7)
im = im2double(rgb2gray(imrgb));

Error in getSift (line 27)
[x, y, ~] = harris(img);

Error in stitch (line 4)
info = getSift(imgFiles);

i go into harris and change the following line

im = im2double(rgb2gray(imrgb));

to
im = im2double((imrgb));

program runs but the final image has a different color than input image, and also doesn’t implement the stitching

10. sunkan says:

okay i twicked the box setting ‘and got it to display right, the stitching works.
now i just need it to remain in the original color palette

11. sunkan says:

Finally got it. did a gray colormap at the end of the figure display

12. Satya says:

While Running this command stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

I am not getting the proper stitched image.

am getting a black image with a portion of the expected output on the right up corner.

Kindly do the needful.

1. PHAM Hoai Vu says:

Like the above, maybe you need to tweak the parameters specified on line 9 of warp.m.
I already mentioned it in this comment

13. Richard says:

Hi Hoai Vu,

I found your code very interesting and it gives nice results.

In my case I am using image sizes of 4000 x 3000 Pixels.
The results of your code with my images are fine, but the algorithm has somewhere downsampled the images.
Therefore the stitched result is Image smaller, I think about a factor 2.

Can you please help me to find the line(s) in the code where I can adapt this to my requirements?

Thanks,
Richard

1. PHAM Hoai Vu says:

Right, it automatically scales the images so that the greatest dimension is 800 pixels.

You can support bigger images by editing stitch.m, change this line:

```info = getSift(imgFiles);
```

into:

```info = getSift(imgFiles, 4000);
```

where 4000 is the greatest dimension of your images.

However note that it will require much more RAM to process such big images. You might end up using all your memory, and be prepared to wait quite long. Matlab in general is not ideal to deal with such scenarios.

14. Richard says:

It works fine! 🙂
Thank you.

15. Hello Sir,

I want stitch multiple images together i tried the code in github ,it’s read more than multi images and do harris, find sift ,computeHomo Info, get homography ,match , dist2, ransac,computeH,Running visualize for every images but in the result of stitching i don’t find the original image .
please can you help me to solve the problem and find a correctly stitching of multi images.

1. PHAM Hoai Vu says:

What does it show in the end?
Is it a black window? If so, you need to change the parameters specified on line 9 of warp.m. I mentioned it in this comment.

1. thanks for the response,this the errors when I put 14 images

Error using repmat
Out of memory. Type HELP MEMORY for your options.

Error in meshgrid (line 58)
xx = repmat(xrow,size(ycol));

Error in vgg_warp_H (line 84)
[U,V] = meshgrid(bb_xmin:bb_xmax,bb_ymin:bb_ymax);

Error in warp (line 15)
Imw = vgg_warp_H(double(images{i}), homoInfo{i}, ‘linear’, bbox);

Error in stitch (line 8)
warp(homoInfo, info.images);

16. Basile says:

Hello, my name is Basile, I’m working for my internship on stich images of the Moon in a laboratory of research at Bangkok University.
Your code works very well, and I thank you very much for your great job. My problem is that I can’t figure out how I can save the image stitched ? The result is displayed, but there is none variables displayed on the working space and I would like to save the result automatically. Do you any ideas ?

1. PHAM Hoai Vu says:

Hi Basile,

In the end of the warp() function (in warp.m), you will see something like this:

```...
figure, imagesc(max(Im1, Im2));
```

where imagesc(max(Im1, Im2)) is the final image you see in the figure. Please feel free to add your code there to save that image.

1. Basile says:

Hello Hoai Vu,

Thanks for the tip, it works very well !
I have another kind of problem now… Each time I add an image, the final image is distorted, and 4 images it getting tough to handle images… So I try to fit the result according to images we have in input. For now, I’m thinking about a code which could do that, but I don’t where I can start…
So if you any ideas about, I’ll be happy to other good advices

Regards.

17. Charles says:

How do you change the code to include more images?

1. PHAM Hoai Vu says:

I think you should look into the computeHomoInfo() function.
I don’t have enough bandwidth for doing this now. If you make any progress or have any question, do keep me informed 😉

18. Veiko says:

Hello I am trying your code, it would be a great help if I could get it to work.

I get an error message that I can not really make something of:

Undefined function ‘rgb2gray’ for input arguments of type ‘uint8’.

Error in harris (line 7)
im = im2double(rgb2gray(imrgb));

Error in getSift (line 27)
[x, y, ~] = harris(img);

Error in stitch (line 4)
info = getSift(imgFiles);

I have R2014a installed and the Imaging toolbox is there, and so is the rgb2gray command. It just does not like a uint8 argument.

Any ideas?
Thanks in advance,
Veiko

1. PHAM Hoai Vu says:

Hi veiko,

What is the dimensions of your image?

Cheers,

19. Sowmya Nagaraj says:

Hi,

How do I use the same code to stitch only 2 images?

1. PHAM Hoai Vu says:

I guess you will need to change the code. But in the mean time you might duplicate the second photo. If you want to stitch A and B, you can feed A, B, B into the function.

20. yzk says:

I have tested your code and I used another set of pictures.
It works well. But, I don’t know how could I save this panorama pictures.
I just wonder where the file pointer of the panorama photos is?
I have found that you had saved the picture successfully using pictures in keble before named stitched.png.

1. PHAM Hoai Vu says:

On line 34 of warp.m, you will see the image being displayed. It is max(Im1, Im2).

You can use the imsave() function in matlab to save the image.

1. yzk says:

I tried to do this, but it seems that Im2w, Im1w and Im3w are not defined. why?

2. PHAM Hoai Vu says:

You put the imsave() command there, and run stitch again, it will work.

3. yzk says:

mask1 = double(sum(Im2w, 3) > 0);
mask1 = cat(3, mask1, mask1, mask1);
Im1 = pyrBlend(Im2w, Im1w, mask1, 5);
figure, imagesc(Im1);

mask1 = double(sum(Im1, 3) > 0);
mask1 = cat(3, mask1, mask1, mask1);
Im2 = pyrBlend(Im1, Im3w, mask1, 5);
figure, imagesc(max(Im1, Im2));
imsave(imagesc(max(Im1, Im2)));

The code in the last line is added by me.
When executing it, it returns error that ‘Im2w’ is not defined.
According to the code, Im1 is linked with Im1w and Im2w, Im2 is linked with Im3w.
But, these three variables cannot be found in other places.
Why could this happen?

4. PHAM Hoai Vu says:

Yes i know, you put the command there, and then run:

stitch({‘cite/cite_a.jpg’, ‘cite/cite_b.jpg’, ‘cite/cite_c.jpg’}

again. You can’t just run warp.m because it needs arguments.

5. PHAM Hoai Vu says:

Oh I am sorry, my bad. You should be using the code in the zip package: https://www.box.com/s/xjnjmrq28o8cmjbxo2yf

Somehow the code in the github repo doesn’t work. I will take a look when I got some spare time.

6. yzk says:

Why does the page shows the file was removed? Can you upload it again?

7. PHAM Hoai Vu says:

It works just fine for me. Did you see the “Download” button?

8. yzk says:

NO. It just shows the following contents:
This shared file or folder link has been removed.
Think it’s a mistake? No worries: Just email the owner or get in touch with Box support. We’re here to help. Meantime, learn more about sharing files on Box – and other features, benefits and solutions – below.

By the way, I did not have a user name of box.com. I will register later and try again.

21. yzk says:

I successfully downloaded the file on box.com. It works well. I wonder whether it could be improved and applied to the one I downloaded previously.

1. PHAM Hoai Vu says:

Great, thanks a lot for the feedback. I will need to update the code on github.
Was getting involved into too many stuff recently.

22. sushanth reddy says:

what are the inputs needed for gethomography function?
I also want to know flow chart of ‘.m’ files?
please help me .

1. sushanth reddy says:

i am using the ransac.m file of your’s it is giving different ‘H’ matrix every time when i run it for the same values of ‘N’ no.of iterations ,inliner dist,inliner cnt.
the values i have given are:
N = 500;
inliner_dist = 2;
inliner_cnt = 0.6;

1. Vu Pham says:

Yes that is expected because in the function, for each iteration it will select a subset of points randomly. You can specify a random seed (using the rng() function) to make it repeatable.

23. sushanth reddy says:

Thanks for the answer.

24. sushanth reddy says:

Then, what are the best set of points that are to be taken.

25. AJ says:

The 3 images in the example are overlap with each other?

1. Vu Pham says:

Yes, there should be some overlap, otherwise it will be impossible to extract the set of matched points.