wiki:ServerCloud1
Last modified 5 weeks ago Last modified on 04/18/12 09:41:31

Table of Contents

  1. Cloud Ubuntu 10.04 Linux server at 1&1
    1. Ubuntu Trac installation and upgrade crash course
      1. Speeding up Trac
    2. Other preparations
      1. Sections of some config files
    3. Migration plan
      1. Trac
      2. Todo
    4. Migration log
    5. Adding more hard disk space for some mounts
    6. Defending the server
    7. Some extra config
    8. Django for ggbtrans

Cloud Ubuntu 10.04 Linux server at 1&1

In February 2012 we subscribed a cloud server at 1&1 Germany for SVN/trac, ggbtrans and the GitHub mirroring development server.

This virtual machine is a Ubuntu 10.04 LTS cloud (64 Bit). Important data:

  • Domain-Name: s402728429.online.de
  • IP: 82.165.143.54
  • 1 CPU, 4 GB RAM, 100 GB hard disk

The following extra packages are installed with apt-get (see /var/log/apt/*.log for more details):

  • mc
  • apache2
  • munin
  • subversion
  • postgresql-8.4
  • python-setuptools
  • python-dev
  • build-essential
  • libapache2-mod-wsgi
  • python-psycopg2 (overwritten with manual compilation, required at all?)
  • libapache2-svn
  • libpq-dev
  • sqlite3
  • ant
  • xmlstarlet
  • openjdk-6-jdk (for javac)
  • python-subversion (maybe not required)
  • zip (for hacking the .egg files)
  • postfix
  • python3-minimal
  • libcurl4-openssl-dev
  • nsis
  • wine1.3 (see CreateMSIonUnix, too)
  • mingw32-binutils
  • webalizer
  • libnet-daemon-perl, libdbi-perl, libplrpc-perl, libdbd-pg-perl
  • rkhunter
  • git-core
  • git-svn

Ubuntu Trac installation and upgrade crash course

Python packages can be installed via two ways: from Debian package and by using easy_install. They will be put into different directories: /usr/lib/python2.6/dist-packages and /usr/local/lib/python2.6/dist-packages. The server side installation will always use the first, but we need the second one for the production use. Another issue is to set the Trac data directory which contains the GeoGebra trac database. Apache can be fine-tuned via the virtual host settings file (/etc/apache2/sites-enabled/default) like this: 

<Location "/trac">
 SetEnv PYTHON_EGG_CACHE "/usr/local/lib/python2.6"
 SetEnv TRAC_ENV "/home/trac"
</Location>

The name of those Python packages which were installed via easy_install, will be put into /usr/local/lib/python2.6/dist-packages/easy-install.pth. Currently these are the following ones:

  • Babel (version 0.9.6)
  • Genshi (version 0.6)
  • Trac (version 0.12.3)
  • TracAccountManager (version 0.2.1dev_r4679)
  • TracXMLRPC-1.1.2_r11148
  • TracTocMacro-11.0.0.3
  • Pygments-1.5 (installed on 2012-03-26, then Apache2 had to be restarted)

The last two packages required special command lines: 

easy_install -Z -U http://trac-hacks.org/svn/xmlrpcplugin/trunk
easy_install http://trac-hacks.org/svn/tocmacro/0.11

The other two packages (Django-1.3.1, psycopg2-2.4.4) were installed via compilation/installation by downloading the source code into /home/install, unpacking and following the recommendations in the INSTALL file. Both packages created their own *.egg-info entries in /usr/local/lib/python2.6/dist-packages. (Django is used for ggbtrans, not for Trac.)

Upgrading is always possible with "easy_install -U <package_name>". It is recommended to upgrade the packages in dependency order.

Trac sometimes complains about various problems after installation or upgrade. It seems the trac-admin tool has a changed syntax for the most recent Trac installations. The following command (or something similar) should help when the SVN repository is not synced with Trac: 

trac-admin /var/trac repository resync '*'

Plugins are simply copied in /var/trac/plugins to the appropriate Python version (2.6 currently). (More info on this at ​http://trac.edgewall.org/wiki/TracPlugins.) So Florian's BurndownGraph and FullProgressWorkflowPlugin .egg files were simply copied to a new name as well.

Finally, restarting Apache can also solve some strange problems...

Speeding up Trac

Other preparations

  • Data has been copied by logging in to www.geogebra.org, then (see the sqlite2sql script below, it seems Luxembourg cannot connect to non-22 ssh port) 
    cd /var/trac/db
sh sqlite2sql
cd /var
rsync -a --rsh=ssh trac root@82.165.143.54:/home
rsync -a --rsh=ssh svn root@82.165.143.54:/home
  • An IP entry was added to the /etc/munin/apache.conf "Allow from" line, apache2 restarted.
  • On the Luxembourg server, an SQL to SQL converter has been created as /var/trac/db/sql2sqlite: 
    zcat trac.db.sql.gz > trac.db.sql && echo ".read trac.db.sql" | sqlite3 trac.db
  • Also for the backway conversion into /var/trac/db/sqlite2sql: 
    sqlite3 trac.db .dump | gzip > trac.db.sql.gz
  • Downloading ​http://trac-hacks.org/svn/sqlitetopgscript/0.10/sqlite2pg into /home/install.
  • As postgres user, "psql" was run, and then "create user trac with password '???'; create database trac with owner = trac;".
  • Modifying /etc/postgresql/8.4/main/pg_hba.conf by enabling trusted connection for host 127.0.0.1/32 (for any user), then postgresql-8.4 restarted.
  • A symlink created for /usr/local/bin/svnlook -> /usr/bin/svnlook (Luxembourg compatibility for the pre-commit hook).
  • A group is created: tracdb (with member www-data).
  • trac2.wsgi is used which has a modified first line because of the different Python interpreter binary.
  • trac.ini has been modified (and copied to trac.ini.new) for the changed SMTP server and the database, and also for the web links.

Sections of some config files

  • postfix 
    ...
myhostname = dev.geogebra.org
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = s16072753.onlinehome-server.info, localhost, localhost.localdomain, localhost
...
  • trac.ini 
    # -*- coding: utf-8 -*-

[account-manager]
account_changes_notify_addresses = zoltan@geogebra.org
force_passwd_change = true
notify_actions = new
password_file = /var/trac/.htpasswd
password_store = HtPasswdStore
user_lock_max_time = 0

[attachment]
max_size = 5242880
render_unsafe_content = false

[browser]
color_scale = True
downloadable_paths = /trunk, /branches/*, /tags/*
hide_properties = svk:merge
intermediate_color = 
intermediate_point = 
newest_color = (255, 136, 136)
oldest_color = (136, 136, 255)
oneliner_properties = trac:summary
render_unsafe_content = false
wiki_properties = trac:description

[changeset]
max_diff_bytes = 10000000
max_diff_files = 0
wiki_format_messages = true

[components]
acct_mgr.* = enabled
acct_mgr.admin.* = enabled
acct_mgr.api.* = enabled
acct_mgr.db.* = enabled
acct_mgr.db.sessionstore = disabled
acct_mgr.htfile.* = enabled
acct_mgr.htfile.abstractpasswordfilestore = disabled
acct_mgr.htfile.htdigeststore = disabled
acct_mgr.http.* = enabled
acct_mgr.http.httpauthstore = disabled
acct_mgr.notification.* = enabled
acct_mgr.notification.accountchangelistener = disabled
acct_mgr.notification.accountchangenotificationadminpanel = disabled
acct_mgr.pwhash.* = enabled
acct_mgr.pwhash.htdigesthashmethod = disabled
acct_mgr.svnserve.* = enabled
acct_mgr.svnserve.svnservepasswordstore = disabled
acct_mgr.web_ui.* = enabled
acct_mgr.web_ui.emailverificationmodule = disabled
acct_mgr.web_ui.loginmodule = disabled
acct_mgr.web_ui.registrationmodule = disabled
burndowngraph.burndownchart.burndownchart = enabled
burndowngraph.utils.googlechartproxy = enabled
tracrpc.* = enabled
tractoc.* = enabled
tractoc.macro.* = enabled

[header_logo]
alt = GeoGebra
height = -1
link = http://dev.geogebra.org/trac
src = http://www.geogebra.org/images/logo.png
width = -1

[inherit]
plugins_dir = 
templates_dir = 

[logging]
log_file = trac.log
log_level = DEBUG
log_type = none

[milestone]
stats_provider = DefaultTicketGroupStatsProvider

[mimeviewer]
enscript_modes = text/x-dylan:dylan:4
enscript_path = enscript
max_preview_size = 262144
mime_map = text/x-dylan:dylan,text/x-idl:ice,text/x-ada:ads:adb
php_path = php
tab_width = 8
treat_as_binary = application/octet-stream,application/pdf,application/postscript,application/rtf

[notification]
admit_domains = 
always_notify_owner = true
always_notify_reporter = true
always_notify_updater = true
ignore_domains = 
mime_encoding = none
smtp_always_bcc = michael@geogebra.org
smtp_always_cc = 
smtp_default_domain = 
smtp_enabled = true
smtp_from = trac@geogebra.org
smtp_from_name = GeoGebra Trac
smtp_password = 
smtp_port = 25
smtp_replyto = trac@geogebra.org
smtp_server = localhost
smtp_subject_prefix = __default__
smtp_user = 
ticket_subject_template = $prefix #$ticket.id: $summary
use_public_cc = false
use_short_addr = false
use_tls = false

[project]
admin = 
admin_trac_url = .
descr = 
footer = Visit the Trac open source project at<br /><a href="http://trac.edgewall.org/">http://trac.edgewall.org/</a>
icon = common/trac.ico
name = GeoGebra
url = http://www.geogebra.org/

[query]
default_anonymous_query = status!=closed&cc~=$USER
default_query = status!=closed&owner=$USER
items_per_page = 100

[report]
items_per_page = 100
items_per_page_rss = 0

[revisionlog]
default_log_limit = 100

[roadmap]
stats_provider = DefaultTicketGroupStatsProvider

[search]
min_query_length = 3

[svn]
branches = trunk,branches/*
tags = tags/*

[ticket]
default_cc = 
default_component = General
default_description = 
default_keywords = 
default_milestone = GeoGebra 4.0
default_owner = 
default_priority = major
default_resolution = fixed
default_severity = 
default_summary = 
default_type = defect
default_version = 
max_comment_size = 262144
max_description_size = 262144
preserve_newlines = default
restrict_owner = true
workflow = ConfigurableTicketWorkflow,TicketWorkflowOpFullProgress

[ticket-custom]
estimatedhours = text
estimatedhours.label = Estimate (h)
estimatedhours.value = 0
progress = select
progress.label = Progress (%)
progress.options = 0|10|20|30|40|50|60|70|80|90|100
progress.value = 0

[ticket-workflow]
accept = new,assigned,accepted,reopened -> accepted
accept.operations = set_owner_to_self
accept.permissions = TICKET_MODIFY
leave = * -> *
leave.default = 1
leave.operations = leave_status
reassign = new,assigned,accepted,reopened -> assigned
reassign.operations = set_owner
reassign.permissions = TICKET_MODIFY
reopen = closed -> reopened
reopen.operations = del_resolution
reopen.permissions = TICKET_CREATE
resolve = new,assigned,accepted,reopened -> closed
resolve.operations = set_resolution,full_progress
resolve.permissions = TICKET_MODIFY

[timeline]
abbreviated_messages = True
changeset_collapse_events = false
changeset_long_messages = false
changeset_show_files = 0
default_daysback = 30
max_daysback = 90
newticket_formatter = oneliner
ticket_show_details = false

[trac]
authz_file = 
authz_module_name = 
auto_reload = False
backup_dir = db
base_url = http://dev.geogebra.org/trac
check_auth_ip = false
# database = sqlite:db/trac.db
database = postgres://trac:???@localhost/trac
debug_sql = False
default_charset = iso-8859-15
htdocs_location = 
ignore_auth_case = false
mainnav = wiki,timeline,roadmap,browser,tickets,newticket,search
metanav = login,logout,prefs,help,about
mysqldump_path = mysqldump
permission_policies = DefaultPermissionPolicy, LegacyAttachmentPolicy
permission_store = DefaultPermissionStore
pg_dump_path = pg_dump
repository_dir = /var/svn/geogebra
repository_type = svn
secure_cookies = False
show_email_addresses = false
show_ip_addresses = false
timeout = 20
use_base_url_for_redirect = False

[wiki]
ignore_missing_pages = true
max_size = 262144
render_unsafe_content =
  • apache2 (000-default) 
    ...
WSGIScriptAlias /trac /home/trac/cgi-bin/trac2.wsgi
<Directory /home/trac/cgi-bin>
 WSGIApplicationGroup %{GLOBAL}
 Order deny,allow
 Allow from all
</Directory>
<Location "/trac/login">
 AuthType Basic
 AuthName "Trac"
 AuthUserFile /home/trac/.htpasswd
 Require valid-user
</Location>
<Directory "/home/trac/cgi-bin">
 AllowOverride None
 Allow from all
 Order allow,deny
 Options None
</Directory>
<Location "/trac">
 SetEnv TRAC_ENV "/home/trac"
 SetEnv PYTHON_EGG_CACHE "/usr/local/lib/python2.6"
</Location>

Migration plan

Trac

  1. Remove /var/svn/geogebra on the new server to avoid conflicts. On www.geogebra.org, run the above commands (5 rows).
  2. "chown -R www-data.www-data /home/svn; chown -R root.tracdb /var/trac/db; chmod 755 /var/trac/db; chmod 644 /var/trac/db/trac.db".
  3. On the new server, run "cd /home/trac/db; sh sql2sqlite".
  4. As postgres user, "psql", and then "drop database trac; create database trac with owner = trac;". If this is not working because of locking, then from Unix command line kill all locking processes.
  5. Run "/home/install/sqlite2pg -e /home/trac -p 'postgres://trac:???@localhost/trac'"
  6. Copy trac.ini.new over trac.ini. Then restart Apache. Try Trac from a web browser via internet.
  7. If needed, trac-admin resync (see above).
  8. Clean up big files on the new server in /home/trac/db. DONE
  9. As postgres user, "psql trac", and then "update wiki set text=replace(text,'​http://dev.geogebra.org/trac','http://dev.geogebra.org/trac');". This will change the wiki links as needed.

Todo

  • geogebra-google@… sync (migrate /var/root/Scripts/sync-svn-with-google from Luxembourg, daily) DONE
  • ggbtrans sync (via git) DONE (but needs additional migration with ggbtrans itself DONE) - ​https://git.wiki.kernel.org/articles/g/i/t/GitSvnSwitch_8828.html was a must to solve this
  • github sync (via git) DONE (but needs additional migration later DONE)
  • autotest autobuild.conf DONE
  • autobuild autobuild.conf DONE (moved to dev)
  • sync-svn-with-webserver in Luxembourg, modify to use dev.geogebra.org repository DONE
  • remove Trac backup from /var/root/Scripts/backup and add such a script for 1&1 DONE
  • general backup for 1&1 DONE
  • redirect from www.geogebra.org/trac to dev.geogebra.org/trac DONE for http: 
    RedirectMatch permanent /trac/(.*) "http://dev.geogebra.org/trac/$1"
RedirectMatch permanent /svn/(.*) "http://dev.geogebra.org/svn/$1"
  • maybe static links from GeoGebra wiki and cms to be changed by scripts, SQL or manually NOT NEEDED
  • auto svnignore DONE

And:

  • https? OK, supported, maybe http should be denied. Also some basic ideas were borrowed from ​http://trac.edgewall.org/ticket/4733 to force https on login in /etc/apache2/sites-available/default: 
    ...
RedirectMatch /trac/login https://dev.geogebra.org/trac/login
</VirtualHost>
  • webalizer to detect spiders and robots? DONE
  • Fix ld, windres, makensis, osslsigncode for installer DONE
  • Fix 70/1025 user/groups
  • Understand why Trac gives this error quite frequently: 
    Trac[main] ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 267, in dispatch
    req.send(output, content_type or 'text/html')
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/api.py", line 417, in send
    self.write(content)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/api.py", line 537, in write
    self._write(data)
IOError: failed to write data

Migration log

  • The trac.db file had to be deleted first.
  • Trac complains about a problem, but it can be ignored: 
    /usr/local/lib/python2.6/dist-packages/TracAccountManager-0.2.1dev_r4679-py2.6.egg/acct_mgr/pwhash.py:13: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5, sha
  • Before loading the data into PostgreSQL, apache had to be stopped. Maybe someone tried to access the SQLite database and the migration script didn't like that.
  • /var/www/images/background-trac.png had to be copied from Luxembourg.
  • In Luxembourg, the trac and svn entries were simply commented in the Apache config files, then "apachectl stop; apachectl start".
  • In Luxembourg, /trac is a simple redirect. /svn is a simple info on the new place.

Adding more hard disk space for some mounts

lvextend -L +5G /dev/mapper/vg00-home
xfs_growfs -d /home

Defending the server

  • /etc/fw drops some spiders by using iptables
  • Added and manually edited some PostgreSQL related statistics for munin with ​pgtester
  • For munin logging, in Apache status.conf "ExtendedStatus On" and 127.0.0.1 to "Allow from" are added. (It seems Apache cannot bind localhost to 127.0.0.1 automatically.)

Some extra config

  • In /etc/aliases, root is forwarded, then newaliases was run.
  • For trac logrotate, the following logrotate file is used (trac.ini also changed for using the new config file): 
    /var/log/trac.log {
  rotate 30
  daily
  compress
  missingok
  notifempty
  create 640 www-data www-data
  postrotate
	/etc/init.d/apache2 restart >/dev/null 2>&1 || true
  endscript
}

Django for ggbtrans

Installed via easy_install.