Fun with spacewalk–repo–sync fail

This is one of those posts for the sake of replicating information that it took me far too long to find online and then only in one place, so maybe someone else finds this and it helps them.

I was trying to mirror a repo with Spacewalk and spacewalk-repo-sync was failing

[root@thing ~]# spacewalk-repo-sync -c  gitlab-centos7-x86_64 
#### Channel label: gitlab-centos7-x86_64 ####
Repo URL: https://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/
ERROR:
 Cannot retrieve repository metadata (repomd.xml) for repository: 
gitlab-centos7-x86_64. Please verify its path and try again
Sync completed.
Total time: 0:00:01
[root@thing ~]#

As messages about failure go, that's not very useful. If you know how rpm repos are organised you can work out that the full url of the repomd.xml file is https://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml but it's not stated in the output. Also there's no information at all about why repomd.xml couldn't be retrieved. Unhelpfully, spacewalk-repo-sync lacks any options that can be used to provide any kind of additional information about exactly what it's doing and what might have gone wrong. (For Google bait I'll include the words verbose, debug and debugging at this point.) I did know that spacewalk-repo-sync worked for other repos that were set up in Spacewalk, so it had to be something about that repo which it didn't like. Eventually I discovered https://www.novell.com/support/kb/doc.php?id=7014059 which includes this gem:

In case the above recommended settings do not solve the issue, please run:

export URLGRABBER_DEBUG=DEBUG
spacewalk-repo-sync -c <channelname> <options> > /var/log/spacewalk-repo-sync-$(date +%F-%R).log 2>&1

So I ran

[root@thing ~]# URLGRABBER_DEBUG=DEBUG spacewalk-repo-sync -c  gitlab-centos7-x86_64 

and got lots of output which revealed the "Cannot retrieve repository metadata" error was caused by an SSL certificate not being trusted. Knowing that I was able to determine there was an issue with the ca-bundle.crt on the server I was running spacewalk-repo-sync on. Once I'd fix that spacewalk-repo-sync was able to mirror the repo in question.

Once I knew that spacewalk-repo-sync uses urlgrabber I could effecively replicate the issue by running

[root@thing ~]# URLGRABBER_DEBUG=DEBUG urlgrabber https://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml

Right now, even knowing that spacewalk-repo-sync uses urlgrabber, I can find barely anything online about how to find out why spacewalk-repo-sync is failing.

Mike Willis : 12 Jul 2016 14:18 |  Tags: Debug Debugging Fail Spacewalk Spacewalk-Repo-Sync Verbose |  Comments (0) |  Report a problem