Commit 904d13e0 authored by Bastien Le Querrec's avatar Bastien Le Querrec
Browse files

update datasets

Fixes #4

* data is not tracked anymore
* datasets are generated directly from opendata
parent 7458f885
Pipeline #2846 failed with stage
in 41 seconds
.vscode/
scripts/data/
json/
data/gen/
\ No newline at end of file
......@@ -15,14 +15,13 @@ build:
stage: build
script:
- make all
- mkdir -p parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- mkdir -p parlementairesjs-${CI_COMMIT_SHORT_SHA}/data/
- cp -L -r icons parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r images parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r json parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r parlementaires.css parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r parlementaires.js parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r demo.html parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r iframe.html parlementairesjs-${CI_COMMIT_SHORT_SHA}/
- cp -L -r data/gen parlementairesjs-${CI_COMMIT_SHORT_SHA}/data/
artifacts:
paths:
- parlementairesjs-${CI_COMMIT_SHORT_SHA}/
......
all: an senat
an:
mkdir -p json/
yq -c -s . data/an/*.yml > json/an.json
yq -c . data/an.yml > json/an-config.json
./scripts/an.sh
senat:
mkdir -p json/
yq -c -s . data/senat/*.yml > json/senat.json
yq -c . data/senat.yml > json/senat-config.json
./scripts/senat.sh
......@@ -158,59 +158,84 @@ You can generate a dataset with `make <dataset-id>`.
## Create your dataset
Datasets are generated from YML files located at `data/<id>/` (one file per MP with `<last name>_<first name>.yml` filename) and their configuration from `data/<id>-config.yml`. If you want to use a script to get a first bunch of data, put it in `scripts/<id>.sh`.
YML MPs files must follow that format:
```yml
id: XXXXXX
last_name: John
first_name: Doe
group: Some group
county: Some county
commissions:
- "A first commission"
- "Another commission"
phone:
- "0123456789"
- "+33600000000"
email:
- "john.doe@example.org"
- "john.doe2@example.org"
twitter: SommeTwitterAccount
facebook: SomeFacebookLink
photo: XXXXXX
Generated datasets are located at `data/<id>/dataset.json` and their configuration is at `data/<id>/config.json`. Each dataset must be generated by a script located at `scripts/<id>.sh`. If you need to have other data in the git tree to include in the generation process, put it in the `misc/<id>/` folder.
Generated datasets are a JSON array like this:
```json
[
{
"id": "XXXXXX",
"last_name": "John",
"first_name": "Doe",
"group": "Some group",
"county": "Some county",
"commissions": [
"A first commission",
"Another commission"
],
"phone": [
"0123456789",
"+33600000000"
],
"email": [
"john.doe@example.org",
"john.doe2@example.org"
],
"twitter": "SommeTwitterAccount",
"facebook": "SomeFacebookLink",
"photo": "XXXXXX"
},
{
"id": "YYYYYY",
"last_name": "John",
"first_name": "Doe",
"group": "Some group",
"county": "Some county",
"commissions": [
"A first commission",
"Another commission"
],
"phone": [
"0123456789",
"+33600000000"
],
"email": [
"john.doe@example.org",
"john.doe2@example.org"
],
"twitter": "SommeTwitterAccount",
"facebook": "SomeFacebookLink",
"photo": "YYYYYY"
}
]
```
Mandatory fields are `last_name`, `first_name`. You should add a phone number or an email address.
Mandatory fields are `last_name`, `first_name`.
Configuration files must follow that format:
```yml
designation:
singular: "singular form"
plural: "plural form"
phone:
local_prefix: "33"
separator: " "
group_size: 2
img:
baseURL: '/images/<id>/'
width: "150px"
height: "192px"
```json
{
"designation": {
"singular": "député·e",
"plural": "député·es"
},
"phone": {
"local_prefix": "33",
"separator": " ",
"group_size": 2
},
"img": {
"baseURL": "/data/gen/an/images/",
"width": "150px",
"height": "192px"
}
}
```
All fields are mandatory. `baseURL` can be a relative URL or with a FQDN. For privacy issues, you should get a copy of MPs' photos in `images/<id>/` and use a relative `baseURL`.
When your dataset is ready, add this entry in the `Makefile`:
```
<id>:
mkdir -p json/
yq -c -s . data/<id>/*.yml > json/<id>.json
yq -c . data/<id>.yml > json/<id>-config.json
```
## License
Source code is MIT licensed. Data may have a different license.
{
"designation": {
"singular": "député·e",
"plural": "député·es"
},
"phone": {
"local_prefix": "33",
"separator": " ",
"group_size": 2
},
"img": {
"baseURL": "/data/gen/an/images/",
"width": "150px",
"height": "192px"
}
}
designation:
singular: "député·e"
plural: "député·es"
phone:
local_prefix: "33"
separator: " "
group_size: 2
img:
baseURL: '/images/an/'
width: "150px"
height: "192px"
id: PA605036
last_name: Abad
first_name: Damien
group: Les Républicains
county: Ain
commissions:
- "Affaires européennes"
- "Finances"
phone:
- "0140636063"
- "0474755448"
email:
- "contact@damien-abad.fr"
- "damien.abad@assemblee-nationale.fr"
twitter:
facebook:
photo: 605036
id: PA719866
last_name: Abadie
first_name: Caroline
group: La République en Marche
county: Isère
commissions:
- "Lois"
phone:
- "0140635047"
email:
- "caroline.abadie@assemblee-nationale.fr"
twitter: AbadieCaroline
facebook: caroline.abadie.16
photo: 719866
id: PA719146
last_name: Acquaviva
first_name: Jean-Félix
group: Libertés et Territoires
county: Haute-Corse
commissions:
- "Lois"
phone:
- "0140630102"
- "0495611589"
email:
- "jean-felix.acquaviva@assemblee-nationale.fr"
twitter: JF_Acquaviva
facebook: jeanfelixacquaviva
photo: 719146
id: PA722038
last_name: Adam
first_name: Damien
group: La République en Marche
county: Seine-Maritime
commissions:
- "Affaires économiques"
phone:
- "0140630215"
email:
- "damien.adam@assemblee-nationale.fr"
twitter: damienadam76
facebook: damienadam76
photo: 722038
id: PA721036
last_name: Adam
first_name: Lénaïck
group: La République en Marche
county: Guyane
commissions:
- "Lois"
phone:
- "0140637444"
email:
- "lenaick.adam@assemblee-nationale.fr"
twitter: LenaickADAM
facebook: lenaick.adam
photo: 721036
id: PA718954
last_name: Ahamada
first_name: Saïd
group: La République en Marche
county: Bouches-du-Rhône
commissions:
- "Finances"
phone:
- "0140633109"
email:
- "said.ahamada@assemblee-nationale.fr"
twitter:
facebook:
photo: 718954
id: PA605963
last_name: Alauzet
first_name: Éric
group: La République en Marche
county: Doubs
commissions:
- "Finances"
phone:
- "0140637351"
email:
- "depute@alauzet.net"
- "eric.alauzet@assemblee-nationale.fr"
twitter: ericalauzet
facebook: ericalauzet
photo: 605963
id: PA724827
last_name: Ali
first_name: Ramlati
group: La République en Marche
county: Mayotte
commissions:
- "Affaires étrangères"
phone:
- "0140633103"
email:
- "ramlati.ali@assemblee-nationale.fr"
twitter:
facebook:
photo: 724827
id: PA720124
last_name: Amadou
first_name: Aude
group: La République en Marche
county: Loire-Atlantique
commissions:
- "Affaires étrangères"
phone:
- "0140630315"
email:
- "permanence@audeamadou.fr"
- "aude.amadou@assemblee-nationale.fr"
twitter: AudeAmadou
facebook: AmadouAude
photo: 720124
id: PA721278
last_name: Anato
first_name: Patrice
group: La République en Marche
county: Seine-Saint-Denis
commissions:
- "Affaires européennes"
- "Affaires économiques"
phone:
- "0140636930"
email:
- "patrice.anato@assemblee-nationale.fr"
twitter:
facebook:
photo: 721278
id: PA721158
last_name: Anglade
first_name: Pieyre-Alexandre
group: La République en Marche
county: Français établis hors de France
commissions:
- "Affaires européennes"
- "Lois"
phone:
- "0140636680"
email:
- "pieyre-alexandre.anglade@assemblee-nationale.fr"
twitter:
facebook:
photo: 721158
id: PA719318
last_name: Anthoine
first_name: Emmanuelle
group: Les Républicains
county: Drôme
commissions:
- "Immunités"
- "Affaires culturelles et éducation"
phone:
- "0140637203"
- "0475483551"
email:
- "emmanuelle.anthoine@assemblee-nationale.fr"
twitter:
facebook: emmanuelle.anthoine.1
photo: 719318
id: PA719100
last_name: Ardouin
first_name: Jean-Philippe
group: La République en Marche
county: Charente-Maritime
commissions:
- "Défense"
phone:
- "0140637337"
- "0140637336"
- "0961670376"
email:
- "jean-philippe.ardouin@assemblee-nationale.fr"
twitter:
facebook: jpardouin
photo: 719100
id: PA720310
last_name: Arend
first_name: Christophe
group: La République en Marche
county: Moselle
commissions:
- "Développement durable"
phone:
- "0140633210"
email:
- "c.arend5706@gmail.com"
- "christophe.arend@assemblee-nationale.fr"
twitter:
facebook:
photo: 720310
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment