Abstract
I've decided to move my public repositories to other forges, and shut down the CGit based public interface to my Git server. This article goes into the reasons for this change.
I've written previously about my personal Git server. Over time, I've also linked to source code repositories hosted on it several times. Like many software projects, my Git server is being forced into change. It stayed the same, but the world didn't, and now it isn't suitable in the same way anymore.
I've decided to move my public repositories to other forges, and shut down the CGit based public interface to my Git server. This article goes into the reasons for this change.
Some background - My Git Server
Like other software developers, I have had many different Git repos over the years. A fair portion of them I wanted to make public, under an open source license. Once upon a time used GitHub for that.
Then in 2018, Microsoft spent a large amount of money to acquire GitHub. Opinions on what this would mean for open source repos were mixed. Some people remembered Microsoft claiming that open source is a cancer, and were worried that GitHub's support for open source repos would suddenly disappear. Other people looked at how Microsoft had changed it's stance to open source in the preceeding years, talking about how they now loved open source, and decided it wouldn't be so bad.
With the uncertainty, it didn't seem like a great time to be too tightly anchored to GitHub, and I had time to take on a new learning project. I signed up for a tiny Linux server from a local cloud hosting company and set up my own personal Git server.
I've gone through a few iterations, and written about it a bit on this blog. The
most relevant aspect to this article is that I didn't only have private
repositories, I also put the public repositories there. If you went to
code.worthe-it.co.za, you would see a frontend running CGit, that allowed
browsing through and cloning all of my open source repos.
The modern realities of running a Git server
In 2021, Microsoft announced what they had been doing with all of those GitHub repositories. GitHub Copilot was announced, a generative AI tool which was trained on GitHub repositories. Since that point, there have been many AI companies building coding models, and they are all hungry for as much data as possible.
Anyone who runs a Git server has run into this problem, sudden traffic spikes,
logs filling up with data that doesn't make sense. It looks like a DDoS attack,
and in many ways it is. It's the AI scrapers coming to take code that they feel
entitled to just because it's public. And the crawlers don't care what they do
to sites in the process, so instead of cloning the repo or looking at one or two
pages like a regular person, they start following every single link. They load
up every single file, at every point in history. They open the git blame page
for every file at every point in history. They diff every combination of
commits. Because these scrapers aren't smart, they're just persistent.
The only area that they seem to have been made smart is in disguising themselves
as regular users. They come from a random assortment of IP addresses. They use
user agent strings that look like regular browsers. They of course completely
disregard robots.txt files. It has been clear from very early on that AI
companies have trouble understanding the idea of consent.
I'm not the authority on how to deal with these. I have seen others deploying tools like Anubis or tar pits.
The decision - turn off the frontend, leave the server
Fighting these scrapers isn't something that I want to spend my time doing. Instead, I've decided to keep my Git server for my private repos, and turn off CGit entirely for the public ones. I've moved most of my public repos to SourceHut, some others were already on Codeberg.
I don't want to claim to be attached to any particular software forge at this point. In fact, I've built functionality into my (now private only) Git server to mirror any code that I push to other forges, so I'm in a good position to move on to the next one if SouceHut suddenly disappears. However, I do love that I can pay SourceHut a reasonable fee, and they will act like I am the customer, and I own the code they host for me.
Some positives out of the whole thing
For a public forge, my CGit frontend was pretty bare-bones. It didn't have any support for accepting contributions. Not even an issue tracker.
SourceHut has built in mailing list support for this purpose. It's a bit unusual these days to use mailing lists for asking questions or submitting patches, but a great consequence is that nobody has to create a SourceHut account to report a bug.
Another thing that I'm enjoying is SourceHut's build servers. I never did get around to setting up any sort of CI infrastructure for my personal Git server.
It is what it is
All of this said, I am sad to need to shut down the public frontend to my forge. It has been there for 8 years. But the world has changed in those 8 years, and my personal hosting sadly needs to keep up.
I hope that SourceHut treats me well!