Add mailman2sympa migration script with my additions

README.md

@@ -23,6 +23,7 @@ it You can also update this README by giving a brief description of your contrib
 Tools for sysadmins.
 
 * [datasources_utils](utils/datasources_utils/README.md): view and test list datasources and custom attributes datasources
+* [mailman2sympa](utils/mailman2sympa/README.md): mailman 2.x migration scripts
 * [sympatoldap](utils/sympatoldap/README.md): creates LDAP entries for every list (and its aliases) whose status is open on the LDAP server
 * [splitting_daemons_logs](utils/splitting_daemons_logs/README.md): split Sympa processus logs into separated files
 * [sqlite2mysql](utils/sqlite2mysql/README.md): attempt to convert sqlite to mysql

utils/mailman2sympa/BUGS

@@ -0,0 +1,5 @@
+BUGS:
+
+The script to load the admins doesn't query the comment_user correctly
+
+probably other stuff too.

utils/mailman2sympa/Changes

@@ -0,0 +1,91 @@
+mailman2sympa 0.7, Konstantin Lepikhov - Nov 2021
+
++ rewrote configuration parsing (use python unpickle)
++ import list info (if exists)
++ don't import if list host_name doesn't match
++ handle unicode names for users/subscribers
++ update import users script for sql scheme changes.
+
+mailman2sympa 0.5, John Bazik - May 2010
+
++ rewrote awk mailman config parsing scripts in perl
++ broke up scripts so that there is one per file generated
++ added translations for more list options
++ added ban_list/blacklist translation
++ dropped old-style subscriber list generation
++ made mailman2sympa.sh a little more concise
++ added mm2s_info script to pull list description info (not integrated)
++ added list_, add_ and remove_members mailman-like scripts
+
+mailman2sympa 0.4, Daniel Black -  March 2009
+
++ fixed lots to make it work with mailman-2.1.9
++ user gecos data now imported
++ mailman user options now respected that bit more
++ admin data now migrated
++ lots more error checking to you don't overwrite stuff unitentionall
++ additional database name to preview the changes in
++ parses mbox archives so no mime hacks are needed
++ email addresses with + now work
+
+mailman2sympa 0.0.2, Andreas de Pretis -  February 2003
+
+
+mailman2sympa.awk
+- -----------------
++) moved to awk/mailman2sympa.awk
++) modified format for aliases (match sympa-generated
+   format)
++) added review, invite, clean_delay_queuemod, domain, footer_type
+   owner, digest, type_archive, expire_task, spam_protection,
+   web_archive_spam_protection, date_epoch, date, profile and
+   the resulting printfs, etc.
++) removed remind_task
++) compatibility with latest sympa-version (3.4.3)
+
+subscribers.awk
+- ---------------
++) moved to awk/subscribers.awk
++) added password-support
++) modified CSV-Format (null instead of _null_, ...)
+
+mailman2sympa.sh
+- ----------------
++) ask user to run loadsubscribers.sh automatically (y/n)
++) created $WDIR if it doesn't exist
++) create CSV-Files in $WDIR/csv as import_users.csv
+   and import_subscribers.csv
++) added DOMAIN=$DOMAIN, SMTPSCRIPTPATH=$SYMPA_SMTPSCRIPTS and 
+   OWNER=$DEFAULT_OWNER when running awk/mailman2sympa.awk
++) added support for ISO-8859-1 Charset (parse $WDIR/lists/$f)
+
+load_subscribers_Pg.pl
+- ----------------------
++) removed because it won't be used by load_subscribers_Pg.sh
+   (which used the loadsubscribers.pl in sympa-home (/bin or /sbin)
+
+load_subscribers_Pg.sh
+- ----------------------
++) removed because of new CSV-Import
+
+mailman2sympa.conf
+- ------------------
++) added 1990-1997 to YEARS
++) default WDIR is /tmp/mailman
++) added DEFAULT_OWNER
++) added CLEANTABLES
++) added HOSTDB and PASSWORDDB
++) added SYMPA_SBIN, SYMPA_LIB and SYMPA_SMTPSCRIPTS
++) default USER and GROUP set to 'sympa' (if you install
+   sympa as described in the documentation, apache must not
+   be the owner of expldir and arcdir
++) added empty line between parameters/settings
+
+general
+- -------
++) moved *.pl and additional sh-scripts to scripts
++) moved awk-scripts to awk
++) moved INSTALL, LICENCE and COPYING to docs
++) moved mailman2sympa.conf to conf
++) slightly modified README and INSTALL to match my changes
+

utils/mailman2sympa/README.md

@@ -0,0 +1,58 @@
+## Presentation
+
+Mailman2sympa is a set of scripts aimed to facilitate the migration
+of mailing list managed by mailman to sympa.
+
+The problem with such scripts is that they are used only once by
+an individual. So it may be difficult to find a permanent
+project manager.
+
+I hope that each user will have the good idea to send back all
+enhancement they would have to apply to the package.
+
+## Features
+
+The version 0.0.3 of the package can do the following tasks:
+
+- create one list in sympa for each list found in the
+  mailman space.
+
+- create the subscribers files et config files for each list
+  with the appropriates attributes, when still relevant.
+
+- creates an aliases file to be concatenated at the bottom
+  of `/etc/aliases`.
+
+- split the archives stored by mailman into the mhonarc
+  archive directory
+
+- in the limited extent, restore the Content-type header
+  when the message is multipart, so that mhonarc may 
+  nicely process attachments.
+
+- load the subscribers files into the database. Can specify
+  a different database `(NEW_DATABASE)` for testing. Change
+  `NEW_DATABASE=DATABASE` for migration
+
+This scripts gave me a good result, but I'm not sure they
+would for you. If you have problems, I can take an hand,
+but as my lists have already been migrated, I donr't have
+anymore a test bed to run the software.
+
+You can contact me at [email protected]
+
+Hope this helps.
+
+Philippe Allart
+
+
+DumperSwitchboard problem - edit $MAILMAN_HOME/bin/dumpdb as follows:
+http://arkiv.netbsd.se/?ml=mailman-users&a=2007-07&t=4791032
+
+### Required packages:
+- gawk
+- libdbd-csv-perl
+- procmail (for archive migration)
+- perl-JSON
+- jq
+- python2 (pickle and json modules)

utils/mailman2sympa/TODO

@@ -0,0 +1,20 @@
+Things that are converted:
+user language preferences,
+user bounce status
+
+Lst config headers and footers
+
+email addresses like
+'=?iso-8859-15?q?=22fred_fr=f6hlich_=28gremlin_list=29?='
+
+
+Some posting rules are not converted correctly.  In particular,
+announce-only lists (what sympa calls a "newsletter") wind up
+moderated for everyone.  To tinker with this, edit the mm2s_config
+script and look for the "send" rule in there.
+
+The mm2s_info script doesn't always work, and isn't integrated into
+the framework.  The info field in mailman is just a field, but in
+sympa it's a separate file.
+
+Virtual domains supprt

utils/mailman2sympa/awk/test_multipart.awk

@@ -0,0 +1,23 @@
+BEGIN { 
+body = 0;
+multipart=0;
+separator="";
+}
+/^$/ {
+	if (body==0) body=1;
+	else if (multipart==1) {
+		print gensub("^--","", "", separator);
+		exit;	
+		}
+}
+/^Content-Type:/ {
+	if (body==1) multipart=1;
+}
+{
+	if (body==1) {
+	  	if($0~"^--.*$") separator=$0;
+		else if (separator=="") separator=$0;
+		}
+
+}
+

utils/mailman2sympa/conf/mailman2sympa.conf.alt

@@ -0,0 +1,127 @@
+#                                                                     
+# IMPORTANT THINGS TO SET                                             
+# DOMAIN                                                              
+# PASSWORDDB                                                          
+# DEFAULT_OWNER                                                       
+
+
+######  Configuration file for Mailman2Sympa ###############
+#
+#  System environment
+#
+# Give below USER an GROUP that sympa is running as
+#
+USER=sympa
+GROUP=sympa
+
+#
+# Where SYMPA has been installed
+#
+SYMPA_SMTPSCRIPTS=/usr/share/sympa/bin
+EXPL=/var/lib/sympa/lists_data
+
+#
+# Where Mailman has been installed
+#
+MAILMAN_HOME=/usr/share/mailman
+MAILMAN_VAR=/var/lib/mailman
+
+#
+# WXhere to put archives (the 'arc_path' value in /etc/wwsympa.conf)
+#
+MHONARC_ARC=/var/lib/sympa/arc
+
+#
+# The domain name of your sever (the right part, after @, of lists adresses)
+DOMAIN=lists.example.com
+
+#
+# Database information
+# Should be the same as in sympa.conf
+#
+SGBD="SQLite"
+DATABASE="/var/lib/sympa/sympa.sqlite"
+# test database - see if everthing loaded ok                                  
+NEW_DATABASE="/var/lib/sympa/sympa-test.sqlite"                                                    
+# for the real migration                                                      
+#NEW_DATABASE="sympa"     
+#USERDB="sympa"
+HOSTDB="localhost"
+#PASSWORDDB=""
+
+#
+# Should all rows in user_table and subscriber_table be deleted before import?
+#
+CLEANTABLES=0
+
+#
+# default owner of lists
+#
+DEFAULT_OWNER="[email protected]"
+
+##################################################################
+# Parameter for the next run of mailman2sympa
+##################################################################
+#
+# Where will be writen temporary and resulting files.
+# After processing, you will find in this directorty some goodies like
+# the email of all users and a semi-column separated file for 
+# subscriptions (not used), and aliases-sympa that you will insert into
+# /etc/aliases
+#
+WDIR=/tmp/mailman
+
+#
+# Do you want keep subscription which have been turned "nomail"
+# You may not want to keep them if you set them as 'nomail'
+# because the adresses were bouncing.
+#
+# If you have users who set thier own setting to nomail due
+# to vacation or other resons they will be dropped if you set
+# this to no. 
+#
+# recomment keeping as yes
+# 
+TAKE_NOMAIL="yes"
+
+#
+# Which years do you want put in archive
+# OK, this syntax could be better, but it has some advantages, and
+# the script is simpler.
+#
+YEARS="1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021"
+
+#
+# Default subscription date
+#
+DEFAULT_DATE="2009-01-01"
+
+#
+# You may want use temporary aliases during 
+# some times for testing
+# Used to generate $WDIR/aliases-sympa
+#
+ALIAS_PREFIX=""
+
+# do you want the aliases to include the domainname? if so say 'yes'
+VIRTUAL_ALIASES="no"
+
+#
+# Do you want pick up mailman archive?
+#
+CONVERT_ARCHIVE="yes"
+
+#
+# Do you want remove and create the sympa archive directory
+# before importing mailman archive?
+#
+CLEAN_ARCHIVE="no"
+
+#
+# Mailman drop Content-Type header, so that 
+# multipart bodies are no more recognized.
+# Do you want try to restore multipart headers?
+#
+RESTORE_MULTIPART="yes"
+
+RESET_PASSWORDS="0"