gien is a tool for exporting the contents of any Github issue tracker to a local mbox file or Maildir directory as email, thereby mapping all issues and comments to email threads.
The structure of the exported email threads mimics Github's email notifications: There is a root message for every issue, and all comments on that issue reply to that root.
The tool can also download images which are referenced using the
markdown syntax  in issues, comments as well
as wiki pages and attach to the correct email message.
We believe that this tool will prove itself helpful when leaving Github, or just for archiving bug trackers.
The exported emails have the content type multipart/alternative and consist of a MIME attachment in text/html format containing a markdown rendering of the Github message text and a text/plain MIME attachment containing the original body text.
Because I'm just using python-markdown for now, some Github-flavoured things could be rendered incorrectly.
Including some image attachments (-d, --download-images option), a typical message has the following structure:
I 1 <no description> [multipa/alternativ, 7bit, 1.5K]
I 2 ├─><no description> [text/html, 7bit, us-ascii, 0.6K]
I 3 └─><no description> [text/plain, 7bit, us-ascii, 0.6K]
A 4 ddd1852a-71c5-11e5-9ec2-0da058cea7ff.png [image/png, base64, 106K]
A 5 e1db00ce-71c5-11e5-8909-bd2991156351.png [image/png, base64, 123K]
When specifying an output mailbox that already exists and contains messages, the program will just append to that file or Maildir. You can use that behaviour to poll the bug tracker of any Github repository for updates for any real email client will help you to delete any dupes.
More intelligent features catering to the purpoes of mirroring issue trackers can be expected in the feature, including partial updates. There is no ETA, though.
Repository wikis can be cloned just like the repos themselves and yield a collection of markdown files corressponding to the individual 'pages'.
When specifying the --wiki option, wiki pages are threaded just like the
issues but with a [WIKI] tag.
By specifying the --threads option, you can increase the number of parallel network requests made when archiving issue trackers. The thread count defaults to 4. This significantly speeds up the archiving process.
Since local processing of the data is very fast and the mirroring of the
wiki relies on first executing a git clone command to fetch the remote
data and then process all the data locally, gien does not implement
parallel processing for wiki data (yet; but it's low priority because
there's little to gain).
Using HTTP basic authentication is not optional at the moment because of the API rate limit, which in pretty much all cases is too low for me.
- python3
- python-pygithub
- python-markdown
- python-progressbar
- python-pygit2
- python-requests
pip3 install gien
On Arch Linux, you can simply install the package
gien from the AUR.
usage: gien [-h] [-I] [-W] [-d] [-i {all,open,closed}] [-l] [-o OUTPUT]
-p PASSWORD -r REPOSITORY [-t THREADS] -u USER
[--mailbox-type {mbox,maildir}]
Export Github issue trackers to local email storage
optional arguments:
-h, --help show this help message and exit
-I, --archive-issues Enable issue archiving.
-W, --archive-wiki Enable wiki archiving. Defaults to off.
-d, --download-images
Enable the downloading of image attachments to issues
and comments.
-i {all,open,closed}, --issues {all,open,closed}
Filter issues by state. Defaults to all.
-l, --labels If the issue has labels, add them to the email
Subject: header. If the issue has been marked as
closed, at a [CLOSED] label to the subject.
-o OUTPUT, --output OUTPUT
Path to the output mbox file or Maildir. Will be
created if it doesn't exist.
-p PASSWORD, --password PASSWORD
Github API authentication: password
-r REPOSITORY, --repository REPOSITORY
Github repository name the issue tracker of which
shall be exported. Example: 2ion/gien
-t THREADS, --threads THREADS
Number of worker threads. Defaults to 4.
-u USER, --user USER Github API authentication: user
--mailbox-type {mbox,maildir}
Specify the mailbox type to use. Defaults to mbox.
Specify -I|--archive-issues to archive issues and/or
-W|--archive-wiki to archive wiki data. If neither of these options
has been given, an empty or unmodified mailbox will be created.
Archive issues from the 2ion/forecast repository in out.mbox:
gien -I --user $user --password $password -r 2ion/forecast -i open -o out.mbox
Archive closed issues as well as the wiki from the mpv-player/mpv
repository, collect referenced images, adding tags and states as labels,
in a Maildir mpv. Because this repository has a lot of content, we
increase the number of worker threads to 10 in order to speed up the
archiving process.
Note that because the pygithub library does not support multithreading yet the initial bootstrap process were all issue metadata is downloaded might take a while before the multithreaded processing part starts.
gien -I -l -W -d -t 10 --mailbox-type maildir -o mpv -u $user -p $password -r mpv-player/mpv
gien -- export Github issue tracker & wiki contents to local email storage
Copyright (C) 2016 Jens John <[email protected]>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.