Detailed explanation of the xxx-STITCH-GRAPH.txt and GES-GSP mode crash (RUN_TYPE=1)

[lin-name]

Dear RussRobin,
Thanks for your great work. I have run this repo on Wins with VS2019 successfully, and get the reslut of the data AANAP-01_skyline in the input-data folder attached in mode GSP(RUN_TYPE = 0) and GES-GSP + SAM(RUN_TYPE = 2), but it crashed in GES-GSP mode (RUN_TYPE=1), below is the output when RUN_TYPE=1:

how can i solve it to get a right output?

Another question, would you please give a detailed explanation of the xxx-STITCH-GRAPH.txt file, especially for the matching graph image edge, what does it mean?

Looking forward for your reply, thanks very much!

[RussRobin]

Hi @lin-name ,

Thank you for your questions.

1. Can you debug the code and provide more details about the GES-GSP crash report? e.g. which line of codes causes the error. I haven't meet such error before but since you can successfully run GSP and OBJ-GSP, the code base and environment seems good.
2. It means that image 0 and image 1 are neighbours, so the algorithms will stitch them together

[lin-name]

@RussRobin ,
Thanks. I have solved the problem. And I will test the repo with my own data.

[lin-name]

Dear @RussRobin,
Sorry to bother you again. I found that the sam function is not used in the repo for the error: fail to open contour_coords.txt in ImageData.cpp. If module sam is missing like above error, will it significantly affect the effect of the image stitching?

I have noticed the sam_test.cpp, we can not find any output information related to contour_coords.txt, so i am wondering whether the contour_coords.txt comes from the sam_test.cpp ? And can you share the *.onnx models used in this file or give a url to download it?

In other words, how can i get the contour_coords.txt and run the sam_test.cpp successfully ?

[RussRobin]

We implemented c++ version of SAM but finally did not use it since the model has to be compressed and is not as good as python implementation. So instead we provide a manual way to generate the txt you mentioned, please see readme:

Run .sln and you will find 0-original.png in the ./ folder. Upload the image to Google Colab and run sam.ipynb to get SAM features and put it in ./ folder.

There are some hyperparameters in SAM inference script that is somehow image specific and you may find it helpful to tune the parameters. If you find the tuning process tedious, it’s also perfectly fine to use our default parameters, which will give you some acceptable, though not optimal, segmentation and stitching results. Also, I do believe this code base can run without SAM and still give reasonable pano.

Hope it makes sense to you.

[lin-name]

Yes, it indeed can run without SAM and still give a reasonable pano.

Also, i have another question, how to get a pano with the center image fixed as the source input. That is the say, for example, if the image_datas[0] is set as the base(standard image or center image as describled in STITCH-GRAPH.txt), we Do NOT make any changes to the image_datas[0], no matter the pose transformation or warping, just deeply copy the input image_datas[0] into the final panoramic image, and then stitch all other images one by one.

Thanks again!

[RussRobin]

That is indeed a good idea! Sorry that I don't think I implemented it in the codes.

[lin-name]

Dear @RussRobin ,

I have another questions, how to keep the size of the output pano? The output pano from this repo may has a zoom, resulting in an abnormal resolution and unequal areas for the same region, like:

as shownd in the image, the two input image's resolution are 15331020, 15321022, but the output pano is 1026*887, it is even smaller than any one of the input.

also like:

the rectangulars in the input and output have unequal areas.

So, how to get the pano with a true dimension(like the areas of the same rectangular pixel region) as in the input. Is there any params or variables to achieve the goal?

Thanks!

[RussRobin]

It is impossible to get the 'true size' of PANO. However, it is definitely possible to keep the size of one image during stitching. To do so, you may want to modify function normalizeVertices in Transform.cpp, which is called in function NISwGSP_Stitching::solve in NISwGSP_Stitching.cpp. The original logic is: the scale and offset of all images are adjusted based on their vertex ranges so that they all fit into the same target_size. So each image is scaled globally. To get the pano that you asked for, all you need to do is to fix the scale of a reference image and only translate the whole image, so that the other images can be put on it in the relative positions found.

Hope it makes sense to you.

[lin-name]

Yes, It is indeed impossible to get the 'true size' of the final PANO. I just want to fix every input image , no scale, then stitch them one by one. As shown above, for example, the door roi in the input image has the same (or almost) areas with that in the final PANO.

Here is the source normalizeVertices in Transform.cpp in your repo,

I'm really sorry, I still don't quite understand how to achieve it.
Could you please demonstrate how to modify this code?

Thanks very much!

[RussRobin]

Sorry that I do not have any available machine with me to test the code myself. However, here is a possible function to fix the size of the first input image that you can build upon:

`
Size2 MultiImages::normalizeVertices(vector<vector>& vertices) {
const vector& ref_verts = vertices[0];

double ref_min_x = 1e9, ref_max_x = -1e9;
double ref_min_y = 1e9, ref_max_y = -1e9;
for (const auto& p : ref_verts) {
    ref_min_x = min(ref_min_x, p.x);
    ref_max_x = max(ref_max_x, p.x);
    ref_min_y = min(ref_min_y, p.y);
    ref_max_y = max(ref_max_y, p.y);
}
double ref_w = ref_max_x - ref_min_x;
double ref_h = ref_max_y - ref_min_y;

double min_x = 1e9, max_x = -1e9;
double min_y = 1e9, max_y = -1e9;
for (auto& vs : vertices) {
    for (auto& p : vs) {
        min_x = min(min_x, p.x);
        max_x = max(max_x, p.x);
        min_y = min(min_y, p.y);
        max_y = max(max_y, p.y);
    }
}

double offset_x = -ref_min_x;
double offset_y = -ref_min_y;
for (auto& vs : vertices) {
    for (auto& p : vs) {
        p.x += offset_x;
        p.y += offset_y;
    }
}

double total_w = max_x - min_x + offset_x;
double total_h = max_y - min_y + offset_y;
return Size2(total_w, total_h);

}
`

Hope it works for you. Fingers crossed.

[Wazxcvb]

Dear RussRobin, Thanks for your great work. I have run this repo on Wins with VS2019 successfully, and get the reslut of the data AANAP-01_skyline in the input-data folder attached in mode GSP(RUN_TYPE = 0) and GES-GSP + SAM(RUN_TYPE = 2), but it crashed in GES-GSP mode (RUN_TYPE=1), below is the output when RUN_TYPE=1:

how can i solve it to get a right output?

Another question, would you please give a detailed explanation of the xxx-STITCH-GRAPH.txt file, especially for the matching graph image edge, what does it mean?

Looking forward for your reply, thanks very much!

Hello, if you are using OpenCV on Windows, can you install it directly using. exe or do you need to recompile it？

[lin-name]

@Wazxcvb I recompile it. I suggest you do that. as we all known, because it will be more easily configuring the opencv_contrib module if we want to use the features detect and match (like sift or ORB etc). By the way, this repo use the vlfeat, so the opencv sift module is not necessary.

[lin-name]

Sorry that I do not have any available machine with me to test the code myself. However, here is a possible function to fix the size of the first input image that you can build upon:

` Size2 MultiImages::normalizeVertices(vector& vertices) { const vector& ref_verts = vertices[0];

double ref_min_x = 1e9, ref_max_x = -1e9;
double ref_min_y = 1e9, ref_max_y = -1e9;
for (const auto& p : ref_verts) {
    ref_min_x = min(ref_min_x, p.x);
    ref_max_x = max(ref_max_x, p.x);
    ref_min_y = min(ref_min_y, p.y);
    ref_max_y = max(ref_max_y, p.y);
}
double ref_w = ref_max_x - ref_min_x;
double ref_h = ref_max_y - ref_min_y;

double min_x = 1e9, max_x = -1e9;
double min_y = 1e9, max_y = -1e9;
for (auto& vs : vertices) {
    for (auto& p : vs) {
        min_x = min(min_x, p.x);
        max_x = max(max_x, p.x);
        min_y = min(min_y, p.y);
        max_y = max(max_y, p.y);
    }
}

double offset_x = -ref_min_x;
double offset_y = -ref_min_y;
for (auto& vs : vertices) {
    for (auto& p : vs) {
        p.x += offset_x;
        p.y += offset_y;
    }
}

double total_w = max_x - min_x + offset_x;
double total_h = max_y - min_y + offset_y;
return Size2(total_w, total_h);

} `

Hope it works for you. Fingers crossed.

@RussRobin THANKS.

I have tried this code. but it is not my goal. I noticed that this repo use the vlfeat and apap module. The vlfeat for the feature point pairs and the apap for the mesh optimize.

1. getFeatureMatches() in the MultiImage.cpp get the point pairs from the source input image.

2. getImageVerticesBySolving(triplets, b_vector) and normalizeVertices(original_vertices) define the size(cols and rows) of the final PANO.

As shown in this two image above, I noticed that the feature_matches is from the source input image(like resolution 800600). and the original_vertices is from the APAP_Stitching and the LeastSquaresConjugateGradient, the feature_matches and the original_vertices are not from the same scale. So, is it possible to convert the original_vertices to the scale of the source input image(like resolution 800600)? That is to say, we can draw and show the new original_vertices on the source input image or even the DOWN_SAMPLE copy of the source input image.

In that case, we can just transform all the input images, without a scale, and stitch them one by one to get the final PANO.

Looking forward for your reply!

[RussRobin]

Are you trying to draw something like this when referring to draw and show the new original_vertices on the source input image or even the DOWN_SAMPLE copy of the source input image? If it is for visualization only, simply scale the output pano seems fine to me.

Also, the mesh grid in our codes are distorted for both input images, so no image can be 'fixed' during stitching. This is different from APAP, since figure 3 of their paper shows that they fix the target image. APAP paper can be found here.

But overall, you can control the scale by setting the maximum x and y value of the original_vertices to the size of one input image, or set mean value of original_vertices to half of input image size.