Project modernization

- continuous integration with travis,
- continuous deployment with openshift,
- add update_score cron,
- sentry for exception tracking,
- continuous documentation build with rtfd,
- configuration made 12factor-ish,
- make use of new django-representatives version for database
  optimization.

.gitignore

 *.sqlite3
 celerybeat-*
 core/static/libs/*
-deploy
 
 # libs
 static/libs
@@ -21,3 +20,6 @@ __pycache__/
 # Installer logs
 pip-log.txt
 pip-delete-this-directory.txt
+.dpl
+db.sqlite
+log/

.openshift/README.md

+The OpenShift `python` cartridge documentation can be found at:
+http://openshift.github.io/documentation/oo_cartridge_guide.html#python
+
+For information about .openshift directory, consult the documentation:
+http://openshift.github.io/documentation/oo_user_guide.html#the-openshift-directory

.openshift/action_hooks/README.md

+For information about action hooks, consult the documentation:
+
+http://openshift.github.io/documentation/oo_user_guide.html#action-hooks

.openshift/action_hooks/build

+export OPENSHIFT_PYTHON_WSGI_APPLICATION=memopol/wsgi.py

.openshift/action_hooks/deploy

+#!/bin/bash
+# This deploy hook gets executed after dependencies are resolved and the
+# build hook has been run but before the application has been started back
+# up again.  This script gets executed directly, so it could be python, php,
+# ruby, etc.
+set -xe
+
+source ${OPENSHIFT_HOMEDIR}app-root/runtime/dependencies/python/virtenv/bin/activate
+
+cat ${OPENSHIFT_REPO_DIR}requirements.txt
+
+pip install -U pip
+
+pip install -r ${OPENSHIFT_REPO_DIR}requirements.txt
+
+# We don't have sentry yet
+# python ${OPENSHIFT_REPO_DIR}manage.py raven test
+
+python ${OPENSHIFT_REPO_DIR}manage.py migrate --noinput
+
+pushd ${OPENSHIFT_DATA_DIR}
+if ! [ -d node ]; then
+    wget https://nodejs.org/dist/v4.2.2/node-v4.2.2-linux-x64.tar.gz
+    tar xvzf node-v4.2.2-linux-x64.tar.gz
+    ln -sfn node-v4.2.2-linux-x64 node
+fi
+popd
+
+pushd ${OPENSHIFT_REPO_DIR}
+if [ -f ${OPENSHIFT_DATA_DIR}sentry ]; then
+    pip install raven
+    ./manage.py raven test
+else
+    echo ${OPENSHIFT_DATA_DIR}sentry does not exist, not setting up raven.
+fi
+
+PATH="${OPENSHIFT_DATA_DIR}node/bin:$PATH"
+HOME=$OPENSHIFT_DATA_DIR
+CI=true
+npm install bower
+npm install
+node_modules/.bin/bower install
+node_modules/gulp/bin/gulp.js less
+mkdir -p wsgi
+./manage.py collectstatic --noinput
+popd
+
+mkdir -p ${OPENSHIFT_DATA_DIR}media
+mkdir -p ${OPENSHIFT_REPO_DIR}wsgi/static/media
+ln -sf ${OPENSHIFT_DATA_DIR}media ${OPENSHIFT_REPO_DIR}wsgi/static/media

.openshift/cron/README.cron

+Run scripts or jobs on a periodic basis
+=======================================
+Any scripts or jobs added to the minutely, hourly, daily, weekly or monthly
+directories will be run on a scheduled basis (frequency is as indicated by the
+name of the directory) using run-parts.
+
+run-parts ignores any files that are hidden or dotfiles (.*) or backup
+files (*~ or *,)  or named *.{rpmsave,rpmorig,rpmnew,swp,cfsaved}
+
+The presence of two specially named files jobs.deny and jobs.allow controls
+how run-parts executes your scripts/jobs.
+   jobs.deny  ===> Prevents specific scripts or jobs from being executed.
+   jobs.allow ===> Only execute the named scripts or jobs (all other/non-named
+                   scripts that exist in this directory are ignored).
+
+The principles of jobs.deny and jobs.allow are the same as those of cron.deny
+and cron.allow and are described in detail at: 
+   http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-Automating_System_Tasks.html#s2-autotasks-cron-access
+
+See: man crontab or above link for more details and see the the weekly/
+     directory for an example.
+
+PLEASE NOTE: The Cron cartridge must be installed in order to run the configured jobs.
+
+For more information about cron, consult the documentation:
+http://openshift.github.io/documentation/oo_cartridge_guide.html#cron
+http://openshift.github.io/documentation/oo_user_guide.html#cron

.openshift/cron/daily/update_database

+#!/bin/bash
+set -x
+
+cmd=$1
+
+cd $OPENSHIFT_REPO_DIR
+export CLEAN=1
+nohup bin/update_all > $OPENSHIFT_LOG_DIR/update_all.log 2>&1 &

.openshift/cron/weekly/README

+Run scripts or jobs on a weekly basis
+=====================================
+Any scripts or jobs added to this directory will be run on a scheduled basis
+(weekly) using run-parts.
+
+run-parts ignores any files that are hidden or dotfiles (.*) or backup
+files (*~ or *,)  or named *.{rpmsave,rpmorig,rpmnew,swp,cfsaved} and handles
+the files named jobs.deny and jobs.allow specially.
+
+In this specific example, the chronograph script is the only script or job file
+executed on a weekly basis (due to white-listing it in jobs.allow). And the
+README and chrono.dat file are ignored either as a result of being black-listed
+in jobs.deny or because they are NOT white-listed in the jobs.allow file.
+
+For more details, please see ../README.cron file.
+

.openshift/cron/weekly/chrono.dat

+Time And Relative D...n In Execution (Open)Shift!

.openshift/cron/weekly/chronograph

+#!/bin/bash
+
+echo "`date`: `cat $(dirname \"$0\")/chrono.dat`"

.openshift/cron/weekly/jobs.allow

+#
+#  Script or job files listed in here (one entry per line) will be
+#  executed on a weekly-basis.
+#
+#  Example: The chronograph script will be executed weekly but the README 
+#           and chrono.dat files in this directory will be ignored.
+#
+#           The README file is actually ignored due to the entry in the
+#           jobs.deny which is checked before jobs.allow (this file).
+#
+chronograph
+

.openshift/cron/weekly/jobs.deny

+#
+#  Any script or job files listed in here (one entry per line) will NOT be
+#  executed (read as ignored by run-parts).
+#
+
+README
+

.openshift/markers/README.md

+For information about markers, consult the documentation:
+
+http://openshift.github.io/documentation/oo_user_guide.html#markers

.travis.yml

 sudo: false
+env:
+  matrix:
+  - DEBUG=True
 language: python
 python:
-  - "2.7"
+- '2.7'
 install:
-  - pip install django
-  - pip install -U pip
-  - pip install -r requirements.txt
-  - cp memopol/config.json.sample memopol/config.json
+- pip install django
+- pip install -r requirements.txt
 before_script:
-  - npm install -g bower
-  - bower install
-  - npm install
+- npm install -g bower
+- bower install
+- npm install
 script:
-  - ./manage.py migrate
-  - node_modules/gulp/bin/gulp.js less
+- ./manage.py migrate
+- node_modules/gulp/bin/gulp.js less
+deploy:
+- provider: openshift
+  user: jamespic@gmail.com
+  password:
+    secure: W7hQDKAtmpOfwLjBuss6NEKqPSrRhsbgH8a8eV+/Oo6HZxMi1mbNFSi+6WRNSs3Cil0ZZV+awoqC61jIzV4oTwEYcy5bv9NWNSY1QO34DECMS5sY00wA0zKhkdsdTr9Pc3TLRp1cw6x2KNCF356FKZojFTRbjtfJ79rqBc5k5ww=
+  app: dev
+  domain: memopol
+  skip_cleanup: true
+  deployment_branch: pr
+  on:
+    repo: political-memory/political_memory
+    branch: pr

bin/lib.sh

+if [ -n "$OPENSHIFT_HOMEDIR" ]; then
+    source ${OPENSHIFT_HOMEDIR}app-root/runtime/dependencies/python/virtenv/bin/activate
+fi
+
+function pipe_download_to_command() {
+    if [ -n "$OPENSHIFT_DATA_DIR" ]; then
+        cd $OPENSHIFT_DATA_DIR
+    fi
+
+    [ -n "$CLEAN" ] && rm -rf $1
+    [ -f "$1" ] || wget http://parltrack.euwiki.org/dumps/$1 || exit 1
+
+    if [ -n "$OPENSHIFT_REPO_DIR" ]; then
+        cd $OPENSHIFT_REPO_DIR
+    fi
+
+    export DJANGO_SETTINGS_MODULE=memopol.settings
+    unxz -c ${OPENSHIFT_DATA_DIR}$1 | $2
+    [ -n "$CLEAN" ] && rm -rf $1
+}

bin/update_all

+#!/bin/bash
+
+bin/update_representatives
+
+# grace time for pg
+sleep 120
+
+bin/update_dossiers
+
+sleep 120
+
+bin/update_votes
+
+sleep 120
+
+bin/update_scores

bin/update_dossiers

+#!/bin/bash
+set -ex
+
+source ${OPENSHIFT_REPO_DIR}bin/lib.sh
+
+pipe_download_to_command ep_dossiers.json.xz parltrack_import_dossiers

bin/update_representatives

+#!/bin/bash
+set -ex
+
+source ${OPENSHIFT_REPO_DIR}bin/lib.sh
+
+pipe_download_to_command ep_meps_current.json.xz parltrack_import_representatives

bin/update_scores

+#!/bin/bash
+set -ex
+
+source ${OPENSHIFT_REPO_DIR}bin/lib.sh
+
+[ -n "$OPENSHIFT_REPO_DIR" ] && cd $OPENSHIFT_REPO_DIR
+./manage.py update_score

bin/update_votes

+#!/bin/bash
+set -ex
+
+source ${OPENSHIFT_REPO_DIR}bin/lib.sh
+
+pipe_download_to_command ep_votes.json.xz parltrack_import_votes

core/templates/admin/index.html

-{% extends "contrib/admin/templates/admin/index.html" %}
-
-{% block branding %}
-    <h1 id="site-name">Administration for Memopol</h1>
-{% endblock %}
-
-{% block content %}
-    
-    
-    {{ block.super }}
-{% endblock %}
-    
-
-    
-
-    

docs/Makefile

+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = _build
+
+# User-friendly check for sphinx-build
+ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
+$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
+endif
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  applehelp  to make an Apple Help Book"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  xml        to make Docutils-native XML files"
+	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+	@echo "  coverage   to run coverage check of the documentation (if enabled)"
+
+clean:
+	rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Memopol.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Memopol.qhc"
+
+applehelp:
+	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
+	@echo
+	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
+	@echo "N.B. You won't be able to view it unless you put it in" \
+	      "~/Library/Documentation/Help or install it in your application" \
+	      "bundle."
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/Memopol"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Memopol"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+latexpdfja:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through platex and dvipdfmx..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."
+
+coverage:
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
+	@echo "Testing of coverage in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/coverage/python.txt."
+
+xml:
+	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
+	@echo
+	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
+
+pseudoxml:
+	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
+	@echo
+	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

docs/conf.py

+# -*- coding: utf-8 -*-
+#
+# Memopol documentation build configuration file, created by
+# sphinx-quickstart on Thu Nov 12 22:42:47 2015.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = []
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'Memopol'
+copyright = u'2015, Laurent Peuch, Mindiell, Arnaud Fabre'
+author = u'Laurent Peuch, Mindiell, Arnaud Fabre'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '1.0'
+# The full version, including alpha/beta/rc tags.
+release = '1.0.0'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ['_build']
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+#keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'alabaster'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None