Image stitching in Matlab

The image below is a panorama taken at the residence where I am living. It is made of 3 different photos which are stitched together using a simple Matlab program that can be downloaded from here, or github.

Image stitching

Image stitching: result

This program is, in fact, a simple programming exercise I have completed recently. The algorithm is quite straightforward:

  1. The input images are fed into a Harris corner detector. In the code, the implementation of Harris detector does not do multi-scale detection, i.e. it is only a single-scale corner detector. It also does not detect interesting points at subpixel accuracy.
  2. Based on detected interesting points, 128-dimensional SIFT descriptors are extracted. The implementation only gives the descriptors which are not rotation-invariant, i.e. it does not try to normalize the dominant orientation of the gradients.
  3. For each pair of 2 images, the feature vectors are matched using the nearest neighborhood algorithm. The distance being used is the Euclidean distance (in 128-dimensional space), and the threshold for matching follows Lowe’s simple approach.
  4. The matched feature vectors are then fed into RANSAC, which computes the homography transformation between two images, given the set of matched feature vectors.
  5. The images are projected (using the estimated homography transformation) in order to get the final result.

The code includes several components which are not mine:

  • harris.m: Harris corner detector by A. Efros
  • find_sift.m: SIFT descriptor by S. Lazebnik
  • and some others.

I only implemented matching and RANSAC, and some other fancy stuffs. Therefore I was able to complete it in several hours, yet the results are quite nice. While running, the program might display some intermediate results, like the set of tentative correspondences (pairs of matched points between two images) and the inliers (estimated in RANSAC). Inliners is always subset of tentative correspondences. A sample result is shown below.

Tentative correspondences and inliners

Tentative correspondences and inliners

Using the script is easy:


> stitch({'cite/cite_a.jpg', 'cite/cite_b.jpg', 'cite/cite_c.jpg'}

The code currently supports 3 images, but extending that to multiple images is easy (by applying the chain rule for homography matrices).

Some other implementations that are available on the web:

Certainly, they are more beautifully implemented than mine, but I am still sharing the code, since it shows that we still can do something nice even with some minimal efforts.

Probably in the next posts, I might come back to some algorithms used in this program, especially RANSAC, a cute algorithm for fitting models.

Advertisements

56 comments

  1. Chào anh Vu, anh có tài liệu nào về Harris Corner Detector anh chia sẽ cho em với em đang làm về nó
    Cảm ơn anh đã đọc tin

  2. Thanks for your awesome & clear direction about how to stitch multiple images together 🙂

    By the way, I experience a issue while executing your source code:

    Read cite/cite_a.jpg: ans =
    600 800 3
    Error using reshape
    To RESHAPE the number of elements must not change.
    Error in colfilt (line 181)
    b(i*mb+brows,j*nb+bcols) = …
    Error in harris (line 28)
    Rmax = colfilt(R,[3 3],’sliding’,nonmax); % find neighbrhood max
    Error in getSift (line 26)
    [x, y, ~] = harris(img);
    Error in stitch (line 6)
    [sifts, siftLoc, images] = getSift(imgFiles);

    May I have your kindly suggestion how to solve this problem? (My platform version of Matlab is R2013b). Please feel free to answer my inquire. Thank you very much:)

    1. Hi Yao-Ling,

      I have just tried the code on github, it works well. Please try again with this command:

      >> stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

      There is no “cite” photos on github, but I included “keble”, so you can give it a try. If it doesn’t work, just let me know.

      Please note that there is a tweak on line 9 of warp.m, where I manually tune the position of the bounding box so that the images are projected correctly. You might want to change those values if you test the code on your own photos.

    2. And I think the code on github already supports more than 3 photos, you can just give it a try.
      I am not sure about the zip package on box.com, it is not well maintained.

      1. Sorry for giving my testing result. Also, thanks for your prompt and kindly reply:)

        I spent some times trying to find the possible solution. Yet, the result currently is still not working due to the same issue I face. (I adapted your suggestion – using the code on ‘github’ instead)

        May I inquire which platform version of MATLAB that you use to process this programme? As I learned, ‘colfilt.m’ is embedded in the ‘Imagetool box’. Not sure whether this is the problem or not… if some parts of code are modified in MATLAB R2013b.

  3. hello sir,
    from which function i have start, harris .m is working then after which have to call ?
    If i wish to call getsitch.m then should be mine input parameter?

    please help me out…

    Thanks and regards

  4. stitch({‘cite/cite_a.jpg’,’cite/cite_b.jpg’,’cite/cite_c.jpg’})

    it giving me error :-

    Error using stitch (line 17)
    Not enough input arguments.

    Error in m1 (line 1)
    stitch({‘cite/cite_a.jpg’,’cite/cite_b.jpg’,’cite/cite_c.jpg’})

    sir please help me out

  5. function [sifts, siftLocations, images] = getSift(imgFiles, maxWidth)

    if i want to call this function , which input arguments should i have to pass??

  6. Sir,
    While running your code which i found at
    https://github.com/phvu/misc/tree/master/imageStitch

    while the function:

    stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

    i am getting this :

    >> m2
    Read keble/keble_a.jpg:
    ans =

    568 720 3

    Running find_sift
    Read keble/keble_b.jpg:
    ans =

    568 720 3

    Running find_sift
    Read keble/keble_c.jpg:
    ans =

    568 720 3

    Running find_sift
    Out of memory. Type HELP MEMORY for your
    options.

    Error in dist2 (line 24)
    n2 = (ones(ncentres, 1) * sum((x.^2)’,
    1))’ + …

    Error in match (line 6)
    dist = dist2(imgFeatures1,
    imgFeatures2);

    Error in getHomography (line 14)
    matches = match(imgFeatures1,
    imgFeatures2, MATCH_THRESHOLD);

    Error in computeHomoInfo (line 21)
    [H, corres_1, corres_2,
    inlierIdx] = …

    Error in stitch (line 6)
    homoInfo =
    computeHomoInfo(info.features,
    info.featureLocations, info.images);

    Error in m2 (line 1)
    stitch({‘keble/keble_a.jpg’,
    ‘keble/keble_b.jpg’,
    ‘keble/keble_c.jpg’})

    1. It simply means what it says: You don’t have enough memory for the program to execute. Please consider running the code on a machine with more RAM.
      In case your machine have enough RAM, then you might need to increase the memory limit of MATLAB.

  7. Hi im having a few problem with my image results.

    i have 3 png images they are already grayed out.

    when i run the program i get the following

    stitch({‘hotel-00.png’,’hotel-01.png’,’hotel-02.png’})
    Read hotel-00.png:
    ans =

    600 800

    Error using rgb2gray>parse_inputs (line 81)
    MAP must be a m x 3 array.

    Error in rgb2gray (line 35)
    X = parse_inputs(varargin{:});

    Error in harris (line 7)
    im = im2double(rgb2gray(imrgb));

    Error in getSift (line 27)
    [x, y, ~] = harris(img);

    Error in stitch (line 4)
    info = getSift(imgFiles);

    i go into harris and change the following line

    im = im2double(rgb2gray(imrgb));

    to
    im = im2double((imrgb));

    program runs but the final image has a different color than input image, and also doesn’t implement the stitching

  8. okay i twicked the box setting ‘and got it to display right, the stitching works.
    now i just need it to remain in the original color palette

  9. While Running this command stitch({‘keble/keble_a.jpg’, ‘keble/keble_b.jpg’, ‘keble/keble_c.jpg’})

    I am not getting the proper stitched image.

    am getting a black image with a portion of the expected output on the right up corner.

    Kindly do the needful.

  10. Hi Hoai Vu,

    I found your code very interesting and it gives nice results.

    In my case I am using image sizes of 4000 x 3000 Pixels.
    The results of your code with my images are fine, but the algorithm has somewhere downsampled the images.
    Therefore the stitched result is Image smaller, I think about a factor 2.

    Can you please help me to find the line(s) in the code where I can adapt this to my requirements?

    Thanks,
    Richard

    1. Right, it automatically scales the images so that the greatest dimension is 800 pixels.

      You can support bigger images by editing stitch.m, change this line:

      info = getSift(imgFiles);
      

      into:

      info = getSift(imgFiles, 4000);
      

      where 4000 is the greatest dimension of your images.

      However note that it will require much more RAM to process such big images. You might end up using all your memory, and be prepared to wait quite long. Matlab in general is not ideal to deal with such scenarios.

  11. Hello Sir,

    I want stitch multiple images together i tried the code in github ,it’s read more than multi images and do harris, find sift ,computeHomo Info, get homography ,match , dist2, ransac,computeH,Running visualize for every images but in the result of stitching i don’t find the original image .
    please can you help me to solve the problem and find a correctly stitching of multi images.

      1. thanks for the response,this the errors when I put 14 images

        Error using repmat
        Out of memory. Type HELP MEMORY for your options.

        Error in meshgrid (line 58)
        xx = repmat(xrow,size(ycol));

        Error in vgg_warp_H (line 84)
        [U,V] = meshgrid(bb_xmin:bb_xmax,bb_ymin:bb_ymax);

        Error in warp (line 15)
        Imw = vgg_warp_H(double(images{i}), homoInfo{i}, ‘linear’, bbox);

        Error in stitch (line 8)
        warp(homoInfo, info.images);

  12. Hello, my name is Basile, I’m working for my internship on stich images of the Moon in a laboratory of research at Bangkok University.
    Your code works very well, and I thank you very much for your great job. My problem is that I can’t figure out how I can save the image stitched ? The result is displayed, but there is none variables displayed on the working space and I would like to save the result automatically. Do you any ideas ?

    1. Hi Basile,

      In the end of the warp() function (in warp.m), you will see something like this:

      ...
      figure, imagesc(max(Im1, Im2));
      

      where imagesc(max(Im1, Im2)) is the final image you see in the figure. Please feel free to add your code there to save that image.

      1. Hello Hoai Vu,

        Thanks for the tip, it works very well !
        I have another kind of problem now… Each time I add an image, the final image is distorted, and 4 images it getting tough to handle images… So I try to fit the result according to images we have in input. For now, I’m thinking about a code which could do that, but I don’t where I can start…
        So if you any ideas about, I’ll be happy to other good advices

        Regards.

    1. I think you should look into the computeHomoInfo() function.
      I don’t have enough bandwidth for doing this now. If you make any progress or have any question, do keep me informed 😉

  13. Hello I am trying your code, it would be a great help if I could get it to work.

    I get an error message that I can not really make something of:

    Undefined function ‘rgb2gray’ for input arguments of type ‘uint8’.

    Error in harris (line 7)
    im = im2double(rgb2gray(imrgb));

    Error in getSift (line 27)
    [x, y, ~] = harris(img);

    Error in stitch (line 4)
    info = getSift(imgFiles);

    I have R2014a installed and the Imaging toolbox is there, and so is the rgb2gray command. It just does not like a uint8 argument.

    Any ideas?
    Thanks in advance,
    Veiko

    1. I guess you will need to change the code. But in the mean time you might duplicate the second photo. If you want to stitch A and B, you can feed A, B, B into the function.

  14. I have tested your code and I used another set of pictures.
    It works well. But, I don’t know how could I save this panorama pictures.
    I just wonder where the file pointer of the panorama photos is?
    I have found that you had saved the picture successfully using pictures in keble before named stitched.png.

    1. On line 34 of warp.m, you will see the image being displayed. It is max(Im1, Im2).

      You can use the imsave() function in matlab to save the image.

      1. mask1 = double(sum(Im2w, 3) > 0);
        mask1 = cat(3, mask1, mask1, mask1);
        Im1 = pyrBlend(Im2w, Im1w, mask1, 5);
        figure, imagesc(Im1);

        mask1 = double(sum(Im1, 3) > 0);
        mask1 = cat(3, mask1, mask1, mask1);
        Im2 = pyrBlend(Im1, Im3w, mask1, 5);
        figure, imagesc(max(Im1, Im2));
        imsave(imagesc(max(Im1, Im2)));

        The code in the last line is added by me.
        When executing it, it returns error that ‘Im2w’ is not defined.
        According to the code, Im1 is linked with Im1w and Im2w, Im2 is linked with Im3w.
        But, these three variables cannot be found in other places.
        Why could this happen?

      2. Yes i know, you put the command there, and then run:

        stitch({‘cite/cite_a.jpg’, ‘cite/cite_b.jpg’, ‘cite/cite_c.jpg’}

        again. You can’t just run warp.m because it needs arguments.

      3. NO. It just shows the following contents:
        This shared file or folder link has been removed.
        Think it’s a mistake? No worries: Just email the owner or get in touch with Box support. We’re here to help. Meantime, learn more about sharing files on Box – and other features, benefits and solutions – below.

        By the way, I did not have a user name of box.com. I will register later and try again.

  15. I successfully downloaded the file on box.com. It works well. I wonder whether it could be improved and applied to the one I downloaded previously.

    1. Great, thanks a lot for the feedback. I will need to update the code on github.
      Was getting involved into too many stuff recently.

  16. what are the inputs needed for gethomography function?
    I also want to know flow chart of ‘.m’ files?
    please help me .

    1. i am using the ransac.m file of your’s it is giving different ‘H’ matrix every time when i run it for the same values of ‘N’ no.of iterations ,inliner dist,inliner cnt.
      the values i have given are:
      N = 500;
      inliner_dist = 2;
      inliner_cnt = 0.6;

      1. Yes that is expected because in the function, for each iteration it will select a subset of points randomly. You can specify a random seed (using the rng() function) to make it repeatable.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s