I think I figured out why registration worked with @Marei33's brain images resampled to isotropic but was weird for original anisotropic data. In the meeting we checked the Elastix step: everything was correct based on logs but it seemed that the images were scaled differently based on the size of the nuclei.
The problem is that in the prealignment step we rotate the image in 3D so the anisotropy axis is rotated too. We don't take it into account and use resolution as if the image is still oriented along the old axes. I think the only way to deal with it is to upsample both fixed and moving image to be isotropic with the best resolution. For example, if moving image has voxel size (10, 5, 5), we should resample to (5, 5, 5). This way we can rotate the object however we want and not lose any details. In the end when we apply the transforms we can downsample the resulting volume to original fixed image resolution.
I think I figured out why registration worked with @Marei33's brain images resampled to isotropic but was weird for original anisotropic data. In the meeting we checked the Elastix step: everything was correct based on logs but it seemed that the images were scaled differently based on the size of the nuclei.
The problem is that in the prealignment step we rotate the image in 3D so the anisotropy axis is rotated too. We don't take it into account and use resolution as if the image is still oriented along the old axes. I think the only way to deal with it is to upsample both fixed and moving image to be isotropic with the best resolution. For example, if moving image has voxel size (10, 5, 5), we should resample to (5, 5, 5). This way we can rotate the object however we want and not lose any details. In the end when we apply the transforms we can downsample the resulting volume to original fixed image resolution.