Skip to content
Extraits de code Groupes Projets

Comparer les révisions

Les modifications sont affichées comme si la révision source était fusionnée avec la révision cible. En savoir plus sur la comparaison des révisions.

Source

Sélectionner le projet cible
No results found

Cible

Sélectionner le projet cible
  • la-quadrature-du-net/memopol/memopol
  • lnclt/political_memory
  • arthur/political_memory
  • agrausem/political_memory
  • periode/memopol
  • Anthony/memopol
  • Porkepix/memopol
  • jaster/memopol
  • luxcem/memopol
  • TAlone/memopol
10 résultats
Afficher les modifications
Affichage de
avec 367 ajouts et 229 suppressions
......@@ -2,11 +2,14 @@
bin/update_representatives
# grace time for pg
sleep 120
sleep 10
bin/update_dossiers
sleep 120
sleep 10
bin/update_proposals
sleep 10
bin/update_votes
#!/bin/bash
set -ex
set -x
source ${OPENSHIFT_REPO_DIR}bin/lib.sh
source bin/lib.sh
pipe_download_to_command ep_dossiers.json.xz parltrack_import_dossiers
parltrack_download_pipe ep_dossiers.json.xz parltrack_import_dossiers
sleep 10
francedata_download_pipe dossiers.json.gz francedata_import_dossiers
#!/bin/bash
set -x
source bin/lib.sh
# Nothing for parltrack - proposals imported with votes
francedata_download_pipe scrutins.json.gz francedata_import_scrutins
#!/bin/bash
set -ex
set -x
source ${OPENSHIFT_REPO_DIR}bin/lib.sh
source bin/lib.sh
pipe_download_to_command ep_meps_current.json.xz parltrack_import_representatives
parltrack_download_pipe ep_meps_current.json.xz parltrack_import_representatives
sleep 10
francedata_download_pipe parlementaires.json.gz francedata_import_representatives
#!/bin/bash
set -ex
set -x
source ${OPENSHIFT_REPO_DIR}bin/lib.sh
source bin/lib.sh
pipe_download_to_command ep_votes.json.xz parltrack_import_votes
parltrack_download_pipe ep_votes.json.xz parltrack_import_votes
sleep 10
parltrack_download_pipe ep_com_votes.json.xz parltrack_import_votes
sleep 10
francedata_download_pipe votes.json.gz francedata_import_votes
if [ "x$1" != "xnoscores" ]; then
sleep 10
refresh_scores
fi
.pagination-block
%nav
%ul.pagination.pagination-sm
- if page_obj.has_previous
%li
%a{'href': '?={queries.urlencode}&page=1',
'aria-label': 'First'}
<i aria-hidden="true" class="fa fa-chevron-circle-left"></i>
%li
%a{'href': '?={queries.urlencode}&page=={page_obj.previous_page_number}',
'aria-label': 'Previous'}
<i aria-hidden="true" class="fa fa-chevron-left"></i>
- for p in page_range
- if p
- if p == page_obj.number
%li.active
%a{'href': ''}
{{ p }}
- else
%li
%a{'href': '?={queries.urlencode}&page=={p}'}
{{ p }}
- if page_obj.has_next
%li
%a{'href': '?={queries.urlencode}&page=={page_obj.next_page_number}',
'aria-label': 'Next'}
<i aria-hidden="true" class="fa fa-chevron-right"></i>
%li
%a{'href': '?={queries.urlencode}&page=={paginator.num_pages}',
'aria-label': 'Last'}
<i aria-hidden="true" class="fa fa-chevron-circle-right"></i>
%div.count
Number of results : {{ paginator.count }}
%br
Number of displayed results :
{{ paginator.per_page }}
(
- for limit in pagination_limits
%a{'href': '?paginate_by={{ limit }}'}
{{ limit }}
- if not forloop.last
\/
)
- if grid_list
- include 'core/blocks/grid-list.html'
Fichier supprimé
......@@ -10,7 +10,7 @@ Authentication in the admin backend
If you haven't already, create a super-administrator
account with command::
./manage.py createsuperuser
memopol createsuperuser
Then use this account to authenticate in the
administration backend located in ``/admin``.
......
......@@ -113,7 +113,7 @@ html_theme = 'alabaster'
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
html_theme_path = ['../memopol_env/local/lib/python2.7/site-packages']
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
......
Deployment on a custom machine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prerequisites
=============
You will need the following:
* A clone of the git repository at ``https://git.laquadrature.net/memopol/memopol.git``
* A dedicated python virtualenv
* a PostgreSQL>=9.2 database (not necessarily on the same machine)
* a WSGI-capable web server
Setup environment
=================
Set the following environment variables::
DJANGO_SETTINGS_MODULE=memopol.settings
Customize settings
==================
Create a copy of the example local settings file::
$ cp src/memopol/local_settings.py.example src/memopol/local_settings.py
Edit ``src/memopol/local_settings.py`` to set directories, database settings and
allowed hosts. Setup your WSGI server to serve:
* Static files from the directory specified in the ``PUBLIC_DIR`` setting to the
``/static`` URL
* The ``src/memopol/wsgi.py`` WSGI application
Initial memopol setup
=====================
From the repository root, install python dependencies (you may want to do that
in a virtualenv)::
$ pip install -U pip setuptools
$ pip install -Ue .
Install client libraries::
$ src/memopol/bin/install_client_deps.sh
Setup the database schema::
$ memopol migrate --noinput
Collect static files::
$ memopol collectstatic --noinput
Memopol should be ready to go.
Updating
========
To update simply pull the repository and run setup commands again::
$ git pull
$ pip install -Ue .
$ src/memopol/bin/install_client_deps.sh
$ memopol migrate --noinput
$ memopol collectstatic --noinput
Data provisionning
==================
Set up two cron jobs:
* One to update data from parliaments, that runs ``bin/update_all``. This
script takes quite some time to run, so you should schedule it once every
night for example
* One to refresh scores, that runs ``memopol refresh_scores``. This one
runs quite quickly (a few seconds), you may want to run it after the update
job has completed (but you can run it more often).
Ensure that cron jobs get the same environment as the application.
Continue to :doc:`administration`.
Deployment on OpenShift
~~~~~~~~~~~~~~~~~~~~~~~
OpenShift is an Open-Source Platform-as-a-Service software by Red Hat. It is
also available in its hosted version known as "OpenShift Online" and the first
three websites ("gears") are free.
Clone the repository
====================
You should fork the project on github and use the fork's clone url. For the
sake of the demo, we'll use the main repository URL::
$ git clone https://github.com/political-memory/political_memory.git
Cloning into 'political_memory'...
remote: Counting objects: 2516, done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 2516 (delta 44), reused 0 (delta 0), pack-reused 2402
Receiving objects: 100% (2516/2516), 4.40 MiB | 79.00 KiB/s, done.
Resolving deltas: 100% (1103/1103), done.
Checking connectivity... done.
$ cd political_memory/
Create your own branch, ie::
$ git checkout -b yourbranch origin/pr
Branch yourbranch set up to track remote branch pr from origin.
Switched to a new branch 'yourbranch'
Create an app on OpenShift
==========================
To deploy the website, use a command like::
$ rhc app-create \
python-2.7 \
cron-1.4 \
postgresql-9.2 \
-a yourappname \
-e OPENSHIFT_PYTHON_WSGI_APPLICATION=memopol/wsgi.py \
--from-code https://github.com/political-memory/political_memory.git \
--no-git
This should create an app on openshift. Other commands would deploy it at once
but in this tutorial we're going to see how to manage it partly manually for
development.
Add the git remote created by OpenShift
=======================================
Add the git remote openshift created for you, you can see it with
``rhc app-show``, ie.::
$ rhc app-show -a yourappname
[snip]
Git URL: ssh://569f5cf500045f6a1839a0a4@yourappname-yourdomain.rhcloud.com/~/git/yourappname.git/
Initial Git URL: https://github.com/political-memory/political_memory.git
SSH: 569f5cf500045f6a1839a0a4@yourappname-yourdomain.rhcloud.com
[snip]
$ git remote add oo_yourappname ssh://569f5cf500045f6a1839a0a4@yourappname-yourdomain.rhcloud.com/~/git/yourappname.git/
Activate OpenShift's git post-recieve hook
==========================================
Activate OpenShift's post-receive hook on your branch::
$ rhc app-configure -a yourappname --deployment-branch yourbranch
Deploy your branch
==================
OpenShift will deploy when it receives commits on the deployment branch, to
deploy just do::
$ git push oo_yourappname yourbranch
If something goes wrong and you want to retry, use the ``rhc app-deploy``
command, ie::
$ rhc app-deploy yourbranch -a yourappname
Data provisionning
==================
To fill up the representatives database table, either wait for the cron script
to be executed, either do it manually::
$ rhc ssh -a yourappname 'cd app-root/repo/ && bin/update_all'
OpenShift is fun, login with ssh and look around if you're curious, you'll be
able to recreate your app without much effort if you break it anyway.
Continue to :doc:`administration`.
Local development tutorial
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. warn:: I reverse-engineered this from the source code I inherited, I might
not be doing the right way nor be able to defend all of technical
decisions.
This tutorial drives through a local installation of the project for
development on Linux. It requires git, a fairly recent version of nodejs (see
:file:`.openshift/action_hooks/deploy` for a way to install it), python2 and
virtualenv.
development on Linux. It requires git, a fairly recent version of python2,
virtualenv and PostgreSQL.
Setup the database
==================
Memopol requires PostgreSQL 9.1 or higher. It used to run with SQLite, too, but
that is no longer the case. It is better to install and configure you're local
PostgreSQL server before starting to install Memopol.
On Debian
---------
To setup you're PostgreSQL database on a debian stable distribution, you can use
the package manager apt::
$ apt install postgresql postgresql-server-dev-9.X
Then you need to create the 'memopol' user and the 'memopol' database::
# To have a root access on Postgres, you need to connect as user 'postgres'
$ su - postgres
$ psql -c "create user memopol with password 'memopol';"
$ psql -c "alter role memopol with createdb;"
$ psql -c "create database memopol with owner memopol;"
$ exit
You're database is now setup for Memopol. You can now launch the 'quickstart.sh'
script to automatically install all the components or do it manually.
In General
----------
Make sure the corresponding user and database exist on your system; the user
will need the 'createdb' permission in order to be able to run tests. To create
them, you may use the following commands::
$ psql -c "create user memopol with password 'memopol';" -U postgres
$ psql -c "alter role memopol with createdb;" -U postgres
$ psql -c "create database memopol with owner memopol;" -U postgres
Automatic Install
=================
There is a quickstart script used and tested manually by some of the
developers. Feel free to try it, but don't worry if it doesn't work for you
then you can do each install step manually, which is recommended because well
that's how you will learn most.
Here's how to try it::
$ git clone gitlab@git.laquadrature.net:memopol/memopol.git
$ cd memopol
$ source bin/quickstart.sh
At this point, you should now run the development server and access to Memopol::
$ memopol runserver
If you want more control or if it doesn't work for you, then follow the steps
below or have a look at what the quickstart script does.
Development helper
===================
You can run the script 'bin/dev.sh' to automaticaly setup some aliases. It works
only with Bash and Zsh.
The script build a custom file named '.memopol.alias' at the root of the project
containing all the aliases for memopol. All the path to the project are build
automatically. A single line is added to your '$HOME/.bashrc' or '$HOME/.zshrc'
to source the aliases.
After execute 'bin/dev.sh' you should close the current terminal and open
another one to have access to the aliases.
There is a quick list of available aliases::
memopol-code : Go into the repository and activate virtualenv and
set Django in debug mode
memopol-launch : Run the development server echo
memopol-update-all : Get all the production data
memopol-refresh-scores : Refresh all scores
.. warning:: If you are using multiple setup of Memopol, it is not recommended to
use this script.
If you need to change the location of the project, you should remove this line
from your .bashrc or .zshrc::
source $PATH_TO_THE_PROJECT/.memopol.alias
Make a virtual environment
==========================
......@@ -28,26 +113,32 @@ Create a python virtual environment and activate it::
$ source memopol_env/bin/activate
Alternatively, use the tox command::
$ tox -e py27
$ source .tox/py27/bin/activate
Clone the repository
====================
You should fork the project on github and use the fork's clone url. For the
sake of the demo, we'll use the main repository URL::
The project is hosted on https://git.laquadrature.net/memopol/memopol
You can get the code with git ::
$ git clone https://github.com/political-memory/political_memory.git
Cloning into 'political_memory'...
remote: Counting objects: 2516, done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 2516 (delta 44), reused 0 (delta 0), pack-reused 2402
Receiving objects: 100% (2516/2516), 4.40 MiB | 79.00 KiB/s, done.
Resolving deltas: 100% (1103/1103), done.
Checking connectivity... done.
$ git clone https://git.laquadrature.net/memopol/memopol
Clonage dans 'memopol'...
remote: Counting objects: 7972, done.
remote: Compressing objects: 100% (2668/2668), done.
remote: Total 7972 (delta 5203), reused 7830 (delta 5099)
Réception d'objets: 100% (7972/7972), 4.88 MiB | 4.73 MiB/s, fait.
Résolution des deltas: 100% (5203/5203), fait.
Vérification de la connectivité... fait.
$ cd political_memory/
$ cd memopol/
Create your own branch, ie::
$ git checkout -b yourbranch origin/pr
$ git checkout -b yourbranch
Branch yourbranch set up to track remote branch pr from origin.
Switched to a new branch 'yourbranch'
......@@ -57,22 +148,22 @@ Install Python dependencies
Then, install the package for development::
$ pip install -e .
Obtaining file:///tmp/political_memory
Collecting django (from political-memory==0.0.1)
Obtaining file:///tmp/memopol
Collecting django (from memopol==0.0.1)
Using cached Django-1.9-py2.py3-none-any.whl
[output snipped for readability]
Installing collected packages: django, sqlparse, django-debug-toolbar, django-pdb, six, django-extensions, werkzeug, south, pygments, markdown, hamlpy, django-coffeescript, ijson, python-dateutil, pytz, political-memory
Running setup.py develop for political-memory
Successfully installed django-1.9 django-coffeescript-0.7.2 django-debug-toolbar-1.4 django-extensions-1.5.9 django-pdb-0.4.2 hamlpy-0.82.2 ijson-2.2 markdown-2.6.5 political-memory pygments-2.0.2 python-dateutil-2.4.2 pytz-2015.7 six-1.10.0 south-1.0.2 sqlparse-0.1.18 werkzeug-0.11.2
Installing collected packages: django, sqlparse, django-debug-toolbar, django-pdb, six, django-extensions, werkzeug, south, pygments, markdown, hamlpy, django-coffeescript, ijson, python-dateutil, pytz, memopol
Running setup.py develop for memopol
Successfully installed django-1.9 django-coffeescript-0.7.2 django-debug-toolbar-1.4 django-extensions-1.5.9 django-pdb-0.4.2 hamlpy-0.82.2 ijson-2.2 markdown-2.6.5 memopol pygments-2.0.2 python-dateutil-2.4.2 pytz-2015.7 six-1.10.0 south-1.0.2 sqlparse-0.1.18 werkzeug-0.11.2
Install client dependencies
===========================
We'll also need to download client libraries::
$ bin/install_client_deps.sh
$ src/memopol/bin/install_client_deps.sh
* Downloading jquery/jquery (2.1.4) from Github...
* Downloading FortAwesome/Font-Awesome (v4.3.0) from Github...
* Downloading lipis/flag-icon-css (0.7.1) from Github...
......@@ -83,45 +174,20 @@ Activate ``DJANGO_DEBUG``
=========================
``DEBUG`` is disabled by default, the development server
won't run properly by default thnen, to enable it export
won't run properly by default then, to enable it export
the ``DJANGO_DEBUG`` variable in the current shell::
$ export DJANGO_DEBUG=True
Run the development server
==========================
Run the development server::
$ ./manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
December 09, 2015 - 21:26:47
Django version 1.8.7, using settings 'memopol.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[09/Dec/2015 21:26:48] "GET / HTTP/1.1" 200 13294
The website is running on ``http://127.0.0.1:8000/``.
Database migrations
===================
The repo comes with a pre-configured SQLite db with sample data so that you can
start hacking right away. However, if you were to use a local postgresql
database ie. with this sort of environment::
Database migrations ensure the database schema is up to date with the project.
If you're not sure, you can run them anyway, they won't do any harm. Use the
following command::
export DJANGO_DATABASE_DEFAULT_NAME=memopol
export DJANGO_DATABASE_DEFAULT_USER=postgres
export DJANGO_DATABASE_DEFAULT_ENGINE=django.db.backends.postgresql_psycopg2
export DJANGO_DEBUG=1
export DJANGO_SETTINGS_MODULE=memopol.settings
Then you could run database migrations::
$ ./manage.py migrate
$ memopol migrate
Operations to perform:
Synchronize unmigrated apps: django_filters, staticfiles, datetimewidget, autocomplete_light, messages, adminplus, compressor, humanize, django_extensions, constance, bootstrap3
Apply all migrations: legislature, votes, database, admin, positions, sessions, representatives, auth, contenttypes, representatives_votes, taggit
......@@ -140,13 +206,34 @@ Then you could run database migrations::
Provision with data
===================
Again, the repo comes with a pre-configured SQLite db with sample data so that
you can start hacking right away. However, you could still reload sample data::
You can load a small data sample for quick setup:
$ memopol loaddata small_sample.json
If you launch memopol for the first time, you need to launch this command :
$ ./manage.py loaddata memopol/fixtures/small_sample.json
$ memopol refresh_scores
Or actual data (takes a while)::
$ bin/update_all
Run the development server
==========================
Run the development server::
$ memopol runserver
Performing system checks...
System check identified no issues (0 silenced).
December 09, 2015 - 21:26:47
Django version 1.8.7, using settings 'memopol.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[09/Dec/2015 21:26:48] "GET / HTTP/1.1" 200 13294
The website is running on ``http://127.0.0.1:8000/``.
Continue to :doc:`administration`.
......@@ -10,12 +10,28 @@ quickstart
<https://www.packtpub.com/application-development/continuous-delivery-and-devops-%E2%80%93-quickstart-guide-second-edition)>`_,
and I bet you'll order a paperback edition for reference !
Testing
=======
Use the ``tox -l`` command to list tests::
$ pip install tox
$ cd memopol/
$ tox -l
Use the ``tox -e`` command to execute a particular test suite::
$ tox -e py27
And use the ``tox`` command without argument to execute all test suites,
exactly like in CI.
Adding random recommendations
=============================
::
$ ./manage.py shell
$ memopol shell
In [1]: from representatives_votes.models import Proposal
In [2]: from votes.models import Recommendation
In [3]: import random
......@@ -32,14 +48,14 @@ test run is.
To create test fixtures for representatives_positions, insert some Position
objects, and reduce the database with::
./manage.py remove_representatives_without_position
./manage.py remove_groups_without_mandate
./manage.py remove_countries_without_group
memopol remove_representatives_without_position
memopol remove_groups_without_mandate
memopol remove_countries_without_group
For representatives_recommendations::
./manage.py remove_proposals_without_recommendation
./manage.py remove_dossiers_without_proposal
./manage.py remove_representatives_without_vote
./manage.py remove_groups_without_mandate
./manage.py remove_countries_without_group
memopol remove_proposals_without_recommendation
memopol remove_dossiers_without_proposal
memopol remove_representatives_without_vote
memopol remove_groups_without_mandate
memopol remove_countries_without_group
docs/img/score_10years.png

3,79 ko

docs/img/score_1year.png

5,5 ko

docs/img/score_exp1k.png

3,1 ko

docs/img/score_exp6.png

4,61 ko

docs/img/score_formula.png

5,99 ko

......@@ -12,11 +12,13 @@ Contents:
:maxdepth: 2
usage
deployment
deploy-custom
administration
scores
api
development
hacker
setup_solr
Indices and tables
==================
......@@ -24,4 +26,3 @@ Indices and tables
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Memopol Scores
~~~~~~~~~~~~~~
Score computation
=================
In Memopol, each representative has a score that shows how their votes match
the recommendations made by administrators on the instance. The total score
of a representative is the sum of their score on each dossier, which in turn is
the sum of their score on each proposal with a recommendation on the dossier.
Each recommendation made by administrators has a weight: a number that tells how
important a specific proposal is. When the representative vote on a proposal
matches the recommendation, their score on the proposal is set to +weight.
Otherwise, it is set to -weight.
Score decay parameters
======================
Memopol allows to set decay parameters so that older votes have a lower
importance in the total representative score. By default, those parameters are
set to values that disable the score decay, so that each vote contributes
identically to the total score no matter how old it is.
The formula used to compute score is the following:
.. image:: img/score_formula.png
Where:
* ``baseScore`` is the base score for the vote computed as explained above;
* ``voteAge`` is the age of the vote in days;
* ``decayNum`` and ``decayDenom`` define the decay rate;
* ``exponent`` define the steepness of the decay.
The corresponding parameters can be set from the Memopol administration
interface (Memopol Settings > Settings); settings keys are ``SCORE_DECAY_NUM``,
``SCORE_DECAY_DENOM``, ``SCORE_EXPONENT``. Additionnaly, the ``SCORE_DECIMALS``
parameter sets how many decimal places are visible when scores are displayed.
The default values for those settings disable score decay by setting
``SCORE_DECAY_NUM`` to 0, ``SCORE_DECAY_DENOM`` and ``SCORE_EXPONENT`` to 1.
If you want to use score decay, start by setting ``SCORE_DECAY_NUM`` to 1, and
``SCORE_DECAY_DENOM`` to the number of days you want votes to matter. The graph
below shows how a score of 1.0 will decay with a 1-year decay (the X axis is in
days).
.. image:: img/score_1year.png
Increasing ``SCORE_DECAY_DENOM`` will make votes matter longer. Here is the
same example but with a 10-year decay.
.. image:: img/score_10years.png
Increasing ``SCORE_EXPONENT`` instead will make the decay cutoff steeper. Here
is an example with a 1-year decay and the exponent set to 6.
.. image:: img/score_exp6.png
Increasing it dramatically will create a brutal cutoff; here is the same example
with the exponent set to 1000:
.. image:: img/score_exp1k.png