docs/hacker.rst

+Hacker guide
+~~~~~~~~~~~~
+
+See a `hacking demo on the Memopol project in some epic
+slides
+<https://slides.com/jamespic/cd-devops/fullscreen#/>`_.
+
+Read about it in `Continuous Delivery and DevOps
+quickstart
+<https://www.packtpub.com/application-development/continuous-delivery-and-devops-%E2%80%93-quickstart-guide-second-edition)>`_,
+and I bet you'll order a paperback edition for reference !
+
+Testing
+=======
+
+Use the ``tox -l`` command to list tests::
+
+    $ pip install tox
+    $ cd memopol/
+    $ tox -l
+
+Use the ``tox -e`` command to execute a particular test suite::
+
+    $ tox -e py27
+
+And use the ``tox`` command without argument to execute all test suites,
+exactly like in CI.
+
+Adding random recommendations
+=============================
+
+::
+
+    $ memopol shell
+    In [1]: from representatives_votes.models import Proposal
+    In [2]: from votes.models import Recommendation
+    In [3]: import random
+
+    In [4]: for p in Proposal.objects.all(): Recommendation.objects.create(proposal=p, recommendation='for', weight=random.randint(1,10))
+
+
+Creating test fixtures
+======================
+
+The largest test fixtures are, the longer it takes to load them, the longer the
+test run is.
+
+To create test fixtures for representatives_positions, insert some Position
+objects, and reduce the database with::
+
+    memopol remove_representatives_without_position
+    memopol remove_groups_without_mandate
+    memopol remove_countries_without_group
+
+For representatives_recommendations::
+
+    memopol remove_proposals_without_recommendation
+    memopol remove_dossiers_without_proposal
+    memopol remove_representatives_without_vote
+    memopol remove_groups_without_mandate
+    memopol remove_countries_without_group

docs/index.rst

+.. Memopol documentation master file, created by
+   sphinx-quickstart on Thu Nov 12 22:42:47 2015.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to Memopol's documentation!
+===================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   usage
+   deploy-custom
+   administration
+   scores
+   api
+   development
+   hacker
+   setup_solr
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/scores.rst

+Memopol Scores
+~~~~~~~~~~~~~~
+
+Score computation
+=================
+
+In Memopol, each representative has a score that shows how their votes match
+the recommendations made by administrators on the instance.  The total score
+of a representative is the sum of their score on each dossier, which in turn is
+the sum of their score on each proposal with a recommendation on the dossier.
+
+Each recommendation made by administrators has a weight: a number that tells how
+important a specific proposal is.  When the representative vote on a proposal
+matches the recommendation, their score on the proposal is set to +weight.
+Otherwise, it is set to -weight.
+
+Score decay parameters
+======================
+
+Memopol allows to set decay parameters so that older votes have a lower
+importance in the total representative score.  By default, those parameters are
+set to values that disable the score decay, so that each vote contributes
+identically to the total score no matter how old it is.
+
+The formula used to compute score is the following:
+
+.. image:: img/score_formula.png
+
+Where:
+
+* ``baseScore`` is the base score for the vote computed as explained above;
+* ``voteAge`` is the age of the vote in days;
+* ``decayNum`` and ``decayDenom`` define the decay rate;
+* ``exponent`` define the steepness of the decay.
+
+The corresponding parameters can be set from the Memopol administration
+interface (Memopol Settings > Settings); settings keys are ``SCORE_DECAY_NUM``,
+``SCORE_DECAY_DENOM``, ``SCORE_EXPONENT``. Additionnaly, the ``SCORE_DECIMALS``
+parameter sets how many decimal places are visible when scores are displayed.
+
+The default values for those settings disable score decay by setting
+``SCORE_DECAY_NUM`` to 0, ``SCORE_DECAY_DENOM`` and ``SCORE_EXPONENT`` to 1.
+
+If you want to use score decay, start by setting ``SCORE_DECAY_NUM`` to 1, and
+``SCORE_DECAY_DENOM`` to the number of days you want votes to matter.  The graph
+below shows how a score of 1.0 will decay with a 1-year decay (the X axis is in
+days).
+
+.. image:: img/score_1year.png
+
+Increasing ``SCORE_DECAY_DENOM`` will make votes matter longer.  Here is the
+same example but with a 10-year decay.
+
+.. image:: img/score_10years.png
+
+Increasing ``SCORE_EXPONENT`` instead will make the decay cutoff steeper.  Here
+is an example with a 1-year decay and the exponent set to 6.
+
+.. image:: img/score_exp6.png
+
+Increasing it dramatically will create a brutal cutoff; here is the same example
+with the exponent set to 1000:
+
+.. image:: img/score_exp1k.png

docs/setup_solr.rst

+Setup Solr with Memopol
+=======================
+
+Solr is used to perform search in the data. Currently, it is used only for
+reprensatative search autocomplete.
+
+This howto will be based on the current stable version of Debian (jessie).
+
+Installation on Debian Stable
+-----------------------------
+
+Solr is in the official repository of debian, you can install it launching::
+
+    # apt install solr-tomcat/stable
+    Package: solr-tomcat
+    Source: lucene-solr
+    Version: 3.6.2+dfsg-5
+    Installed-Size: 65,5 kB
+    Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
+    Depends: solr-common (= 3.6.2+dfsg-5), tomcat7
+    Conflicts: solr-jetty, solr-tomcat6
+    Homepage: http://lucene.apache.org
+    Section: java
+    Priority: optional
+    Download-Size: 8 598 B
+    APT-Sources: http://ftp.fr.debian.org/debian/ jessie/main amd64 Packages
+    Description: Enterprise search server based on Lucene3 - Tomcat integration
+     Solr is an open source enterprise search server based on the Lucene
+     Java search library, with XML/HTTP and JSON APIs, hit highlighting,
+     faceted search, caching, replication, and a web administration
+     interface. It runs in a Java servlet container such as Tomcat.
+     .
+     This package provides the Tomcat integration files for Solr.
+
+
+By default, the solr server is listen on localhost:8080. Memopol is configured
+to use this addess by default, in production. If you install a newer version of
+Solr (6.5.1 is the latest release), you should update the settings.py::
+
+    HAYSTACK_CONNECTIONS = {
+        'default': {
+            'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
+            'URL': 'http://127.0.0.1:8080/solr',
+        },
+    }
+
+Configure Solr
+---------------
+
+You can use the default Solr configuration to use it, You just need to generate
+the schema.xml file and copy it in `/etc/solr/`. To generate the schema.xml
+file::
+
+    $ memopol build_solr_schema > schmema.xml
+
+    ## Copy the schema.xml file into /etc/solr/conf
+    $ sudo cp schema.xml /etc/solr/conf/
+
+    ## Restart tomcat
+    $ sudo /etc/init.d/tomcat7 restart
+
+Last step, you need to build the index by using::
+
+    $ memopol rebuild_index
+
+The solr is now setup and production ready.
+
+Populate data in Solr
+---------------------
+
+Django-haystack plugin offers two ways to populate data. The first way is by
+using a cron job to update the index, the other way is to use `Dango signals to
+update / delete datas <https://django-haystack.readthedocs.io/en/v2.6.0/signal_processors.html>`_.
+
+For now, Haystack is not configured to be used with Django signals. It is 
+necessary to add a cron job to update the index ::
+
+    $ memopol update_index
+
+`More informations about cron with Solr <http://django-haystack.readthedocs.io/en/v2.6.0/searchindex_api.html?highlight=cron#keeping-the-index-fresh>`_

docs/usage.rst

+User guide
+~~~~~~~~~~
+
+Authentication in the admin backend
+===================================
+
+As a content-editor, you should be able to connect to the administration
+interface with the credentials and link that your administrator sent you. If
+they haven't, please email them with a request and eventually a link to
+:doc:`administration`.
+
+Managing representative positions
+=================================
+
+Our dear representatives make promises to be elected. These can be submitted by
+visitors on the front-end so one of your tasks is to review them and publish
+them if they are appropriate.
+
+In "Positions -> Position -> Change" (url should be
+``/admin/positions/position/``), above the list table, click the "Published"
+select box and choose the "Published: No" option, then click "Search".
+Alternatively, you may bookmark
+``/admin/positions/position/?published__exact=0``.
+
+Click a "Position" and if it's appropriate then check the "Published" checkbox
+and click "Save". The position will now appear in the representative detail
+page.
+
+Managing vote recommendations
+=============================
+
+A recommendation represents the vote we want representatives to make on a
+proposal: representatives doing the "recommended" vote will have their score
+increased, others will have their score decreased. Some recommendations may be
+more important than others, you can change the number of score points a
+recommendation is worth by changing its "weight" (must be a positive integer).
+
+In "Votes -> Recommendations -> Change" (url should be
+``/admin/votes/recommendation/``), you can create, update or remove
+recommendations.
+
+Your change won't be visible on score as soon as you make it.
+
+Data updates
+============
+
+Data updates daily:
+
+- representatives details are updated daily,
+- dossiers are synchronised daily,
+- proposals are synchronised daily, when proposals are synchronised, they
+  become available to use in "Recommendations".
+- votes and scores are synchronised daily for proposals which have a
+  "Recommendation",
+
+This means that if a proposal arrives today in the database, you may add a
+recommendation for it. The next day, votes will be imported and scores will be
+updated as well.

legislature/admin.py

-from __future__ import absolute_import
-
-from django.contrib import admin
-
-from representatives.models import Email, WebSite, Address, Phone, Country
-from .models import MemopolRepresentative
-
-
-class EmailInline(admin.TabularInline):
-    model = Email
-    extra = 0
-
-
-class WebsiteInline(admin.TabularInline):
-    model = WebSite
-    extra = 0
-
-
-class AdressInline(admin.StackedInline):
-    model = Address
-    extra = 0
-
-
-class PhoneInline(admin.TabularInline):
-    model = Phone
-    extra = 0
-
-
-class CountryInline(admin.TabularInline):
-    model = Country
-    extra = 0
-    
-    
-class MemopolRepresentativeAdmin(admin.ModelAdmin):
-    list_display = ('full_name', 'country', 'score')
-    search_fields = ('first_name', 'last_name', 'birth_place')
-    list_filter = ('gender', 'active')
-    inlines = [
-        PhoneInline,
-        EmailInline,
-        WebsiteInline,
-        AdressInline,
-    ]
-
-    
-# class MandateAdmin(admin.ModelAdmin):
-    # list_display = ('representative', 'group', 'role', 'constituency', 'begin_date', 'end_date', 'active')
-    # search_fields = ('representative', 'group', 'constituency')
-    # list_filter = ('role',)
-
-
-# admin.site.register(Representative, RepresentativeAdmin)
-admin.site.register(MemopolRepresentative, MemopolRepresentativeAdmin)
-# admin.site.register(Country)
-
-# admin.site.register(MemopolMandate, MandateAdmin)
-
-# admin.site.register(MemopolGroup)
-# admin.site.register(Constituency)

legislature/management/commands/update_memopol_representatives.py

-from __future__ import absolute_import
-
-from django.db import transaction
-from django.core.management.base import BaseCommand
-
-import pyprind
-
-import representatives.models
-import legislature.models
-
-
-class Command(BaseCommand):
-    
-    @transaction.atomic
-    def handle(self, *args, **options):
-        bar = pyprind.ProgBar(representatives.models.Representative.objects.all().count())
-        for i, representative in enumerate(representatives.models.Representative.objects.all()):
-            try:
-                memopol_representative = legislature.models.MemopolRepresentative.objects.get(
-                    representative_remote_id = representative.remote_id
-                )
-            except legislature.models.MemopolRepresentative.DoesNotExist:
-                memopol_representative = legislature.models.MemopolRepresentative(
-                    representative_remote_id = representative.remote_id
-                )
-            memopol_representative.representative_ptr_id = representative.pk
-            
-            memopol_representative.__dict__.update(representative.__dict__)
-            memopol_representative.save()
-            memopol_representative.update_country()
-            
-            bar.update()
-        
-        for i, group_item in enumerate(representatives.models.Group.objects.all()):
-            memopol_group, _ = legislature.models.MemopolGroup.objects.get_or_create(
-                group = group_item
-            )
-            memopol_group.__dict__.update(group_item.__dict__)
-            memopol_group.update_active()
-            memopol_group.save()
-        

legislature/management/commands/update_representatives_score.py

-import pyprind
-
-from django.core.management.base import BaseCommand
-from legislature.models import MemopolRepresentative
-
-class Command(BaseCommand):
-    
-    def handle(self, *args, **options):
-        bar = pyprind.ProgBar(MemopolRepresentative.objects.all().count())
-        for representative in MemopolRepresentative.objects.all():
-            representative.update_score()
-            bar.update(item_id = str(representative))

legislature/migrations/0001_initial.py

-# -*- coding: utf-8 -*-
-from __future__ import unicode_literals
-
-from django.db import models, migrations
-
-
-class Migration(migrations.Migration):
-
-    dependencies = [
-        ('representatives', '0001_initial'),
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='MemopolGroup',
-            fields=[
-                ('group', models.OneToOneField(parent_link=True, primary_key=True, serialize=False, to='representatives.Group')),
-                ('active', models.BooleanField(default=False)),
-            ],
-            options={
-            },
-            bases=('representatives.group',),
-        ),
-        migrations.CreateModel(
-            name='MemopolRepresentative',
-            fields=[
-                ('representative_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='representatives.Representative')),
-                ('representative_remote_id', models.CharField(unique=True, max_length=255)),
-                ('score', models.IntegerField(default=0)),
-                ('country', models.ForeignKey(to='representatives.Country', null=True)),
-            ],
-            options={
-            },
-            bases=('representatives.representative',),
-        ),
-    ]

legislature/models.py

-# coding: utf-8
-
-# This file is part of memopol.
-#
-# memopol is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of
-# the License, or any later version.
-#
-# memopol is distributed in the hope that it will
-# be useful, but WITHOUT ANY WARRANTY; without even the implied
-# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU General Affero Public
-# License along with django-representatives.
-# If not, see <http://www.gnu.org/licenses/>.
-#
-# Copyright (C) 2015 Arnaud Fabre <af@laquadrature.net>
-
-from datetime import datetime
-
-from django.db import models
-from django.core.exceptions import ObjectDoesNotExist
-
-from representatives.models import Representative, Group, Country
-from representatives_votes.models import Vote
-
-class MemopolRepresentative(Representative):
-    
-    representative_remote_id = models.CharField(max_length=255, unique=True)
-    country = models.ForeignKey(Country, null=True)
-    score = models.IntegerField(default=0)
-    
-    def update_score(self):
-        score = 0
-        for vote in self.representative.votes.all():
-            proposal = vote.m_proposal
-            if proposal.recommendation:
-                recommendation = proposal.recommendation
-                if vote.position != recommendation.recommendation:
-                    score -= recommendation.weight
-                else:
-                    score += recommendation.weight
-        self.score = score
-        self.save()
-
-    def update_country(self):
-        # Create a country if it does'nt exist
-        # The representative's country is the one associated
-        # with the last 'country' mandate
-        try:
-            country_mandate = self.mandates.filter(
-                group__kind='country'
-            ).order_by('-begin_date')[0:1].get()
-
-            country, _ = Country.objects.get_or_create(
-                name=country_mandate.group.name,
-                code=country_mandate.group.abbreviation
-            )
-            self.country = country
-            self.save()
-
-        except ObjectDoesNotExist:
-            self.country = None
-            self.save()
-
-
-    # @property
-    # def votes(self):
-        # return Vote.objects.filter(
-            # representative_remote_id = self.remote_id
-        # )
-
-    def active_mandates(self):
-        return self.mandates.filter(
-            end_date__gte=datetime.now()
-        )
-
-    def former_mandates(self):
-        return self.mandates.filter(
-            end_date__lte=datetime.now()
-        )
-    
-    def current_group_mandate(self):
-        return self.mandates.get(
-            end_date__gte=datetime.now(),
-            group__kind='group'
-        )
-
-class MemopolGroup(Group):
-    group = models.OneToOneField(
-        Group,
-        parent_link = True
-    )
-    
-    active = models.BooleanField(default=False)
-
-    def update_active(self):
-        self.active = False
-        for mandate in self.mandates.all():
-            if mandate.end_date > datetime.date(datetime.now()):
-                self.active = True
-                break
-        self.save()
-
-

legislature/templates/legislature/groups_list.haml

--# List representatives' groups
-  
-- extends 'base.html'
-- load by_group_url
-
-- block content
-  %ul
-    - for group in groups
-      %li
-        %a{'href': '{{ group | by_group_url }}'}
-          - if group.abbreviation
-            {{ group.abbreviation }} . 
-          {{ group.name }}

legislature/templates/legislature/representative_block.haml

-- load by_group_url
-
-%td
-  %a{'href': "{% url 'legislature:representative_view_by_name' representative.full_name %}"} 
-    %img{'src': '={representative.photo}', 'width': '80'}/
-
-%td
-  %a{'href': "{% url 'legislature:representative_view_by_name' representative.full_name %}"} 
-    ={representative.full_name}
-
-%td
-  %a{'href': "{% url 'legislature:representatives_by_group' group_kind='country' search=representative.country.code %}"}
-    ={representative.country.name}
-
-%td
-  %a{'href': "{{ representative.current_group_mandate|by_group_url }}"}
-    ={representative.current_group_mandate.group.abbreviation}

legislature/templates/legislature/representative_list.haml

-- extends "base.html"
-
-- block content
-
-  - include 'legislature/search.html'
-
-  %p
-    Number of representatives: {{ representative_num  }}
-
-  %table
-    - for representative in representatives
-      %tr
-        - include 'legislature/representative_block.html'
-
-  .pagination
-    %span.step-links
-      - if representatives.has_previous
-        %a{'href': '?={queries.urlencode}&page=={representatives.previous_page_number}'} previous
-
-      %span.current
-        Page ={representatives.number} of ={representatives.paginator.num_pages}
-
-      - if representatives.has_next
-        %a{'href': '?={queries.urlencode}&page=={representatives.next_page_number}'} next

legislature/templates/legislature/representative_view.haml

--# List representatives
-  
-- extends 'base.html'
-- load by_group_url
-
-- block content
-  %p{:style => "float: left"}
-    %img{:src => "{{ representative.photo }}"}/
-    
-  %h1= representative.full_name
-
-  %p
-    %strong
-      %a{:href => "{{ representative.current_group_mandate|by_group_url }}"}
-        {{ representative.current_group_mandate.role }} of
-        {{ representative.current_group_mandate.group.name }}
-
-  %p
-    Born in {{ representative.birth_place }} the
-    {{ representative.birth_date }} ({{ representative.get_gender_display }})
-
-  %p= representative.country.name
-
-  -# Mandates
-  
-  %h2{:style => "clear: both"} Mandates
-  
-  - for mandate in representative.active_mandates
-    .mandate
-      %h3
-        {{ mandate.group.name }}
-        %small= mandate.role
-      %p
-        {{ mandate.begin_date }} to {{ mandate.end_date }} <br>
-        %a{:href => "{{ mandate|by_group_url }}"}
-          %strong {{ mandate.group.kind }} :
-          %em {{ mandate.group.name }} ({{ mandate.group.abbreviation }}) <br>
-          Constituency : {{ mandate.constituency.name }} <br>
-  %hr/
-
-  -# Former mandates
-    
-  %h2{:style => "clear: both"} Former mandates
-  
-  - for mandate in representative.former_mandates
-    .mandate
-      %h3
-        {{ mandate.group.name }}
-        %small= mandate.role
-      %p
-        {{ mandate.begin_date }} to {{ mandate.end_date }} <br>
-        %a{:href => "{{ mandate|by_group_url }}"}
-          %strong {{ mandate.group.kind }} :
-          %em {{ mandate.group.name }} ({{ mandate.group.abbreviation }}) <br>
-          Constituency : {{ mandate.constituency.name }} <br>
-  %hr/

legislature/templates/legislature/search.html

-<form action="" method="get">
-  {% csrf_token %}
-  <label for="search">Search : </label>
-  <input id="search" type="text" name="search">
-  <input type="submit" value="Go">
-</form>