update datasets

Fixes #4

* data is not tracked anymore
* datasets are generated directly from opendata

.gitignore

 .vscode/
 scripts/data/
 json/
+data/gen/
\ No newline at end of file

.gitlab-ci.yml

@@ -15,14 +15,13 @@ build:
   stage: build
   script:
     - make all
-    - mkdir -p parlementairesjs-${CI_COMMIT_SHORT_SHA}/
+    - mkdir -p parlementairesjs-${CI_COMMIT_SHORT_SHA}/data/
     - cp -L -r icons parlementairesjs-${CI_COMMIT_SHORT_SHA}/
-    - cp -L -r images parlementairesjs-${CI_COMMIT_SHORT_SHA}/
-    - cp -L -r json parlementairesjs-${CI_COMMIT_SHORT_SHA}/
     - cp -L -r parlementaires.css parlementairesjs-${CI_COMMIT_SHORT_SHA}/
     - cp -L -r parlementaires.js parlementairesjs-${CI_COMMIT_SHORT_SHA}/
     - cp -L -r demo.html parlementairesjs-${CI_COMMIT_SHORT_SHA}/
     - cp -L -r iframe.html parlementairesjs-${CI_COMMIT_SHORT_SHA}/
+    - cp -L -r data/gen parlementairesjs-${CI_COMMIT_SHORT_SHA}/data/
   artifacts:
     paths:
     - parlementairesjs-${CI_COMMIT_SHORT_SHA}/

Makefile

 all: an senat
 
 an:
-	mkdir -p json/
-	yq -c -s . data/an/*.yml > json/an.json
-	yq -c . data/an.yml > json/an-config.json
+	./scripts/an.sh
 
 senat:
-	mkdir -p json/
-	yq -c -s . data/senat/*.yml > json/senat.json
-	yq -c . data/senat.yml > json/senat-config.json
+	./scripts/senat.sh

README.md

@@ -158,59 +158,84 @@ You can generate a dataset with `make <dataset-id>`.
 
 ## Create your dataset
 
-Datasets are generated from YML files located at `data/<id>/` (one file per MP with `<last name>_<first name>.yml` filename) and their configuration from `data/<id>-config.yml`. If you want to use a script to get a first bunch of data, put it in `scripts/<id>.sh`.
-
-YML MPs files must follow that format:
-
-```yml
-id: XXXXXX
-last_name: John
-first_name: Doe
-group: Some group
-county: Some county
-commissions:
-- "A first commission"
-- "Another commission"
-phone:
-- "0123456789"
-- "+33600000000"
-email:
-- "john.doe@example.org"
-- "john.doe2@example.org"
-twitter: SommeTwitterAccount
-facebook: SomeFacebookLink
-photo: XXXXXX
+Generated datasets are located at `data/<id>/dataset.json` and their configuration is at `data/<id>/config.json`. Each dataset must be generated by a script located at `scripts/<id>.sh`. If you need to have other data in the git tree to include in the generation process, put it in the `misc/<id>/` folder.
+
+Generated datasets are a JSON array like this:
+
+```json
+[
+  {
+    "id": "XXXXXX",
+    "last_name": "John",
+    "first_name": "Doe",
+    "group": "Some group",
+    "county": "Some county",
+    "commissions": [
+      "A first commission",
+      "Another commission"
+    ],
+    "phone": [
+      "0123456789",
+      "+33600000000"
+    ],
+    "email": [
+      "john.doe@example.org",
+      "john.doe2@example.org"
+    ],
+    "twitter": "SommeTwitterAccount",
+    "facebook": "SomeFacebookLink",
+    "photo": "XXXXXX"
+  },
+  {
+    "id": "YYYYYY",
+    "last_name": "John",
+    "first_name": "Doe",
+    "group": "Some group",
+    "county": "Some county",
+    "commissions": [
+      "A first commission",
+      "Another commission"
+    ],
+    "phone": [
+      "0123456789",
+      "+33600000000"
+    ],
+    "email": [
+      "john.doe@example.org",
+      "john.doe2@example.org"
+    ],
+    "twitter": "SommeTwitterAccount",
+    "facebook": "SomeFacebookLink",
+    "photo": "YYYYYY"
+  }
+]
 ```
 
-Mandatory fields are `last_name`, `first_name`. You should add a phone number or an email address.
+Mandatory fields are `last_name`, `first_name`.
 
 Configuration files must follow that format:
 
-```yml
-designation:
-  singular: "singular form"
-  plural: "plural form"
-phone:
-  local_prefix: "33"
-  separator: " "
-  group_size: 2
-img:
-  baseURL: '/images/<id>/'
-  width: "150px"
-  height: "192px"
+```json
+{
+  "designation": {
+    "singular": "député·e",
+    "plural": "député·es"
+  },
+  "phone": {
+    "local_prefix": "33",
+    "separator": " ",
+    "group_size": 2
+  },
+  "img": {
+    "baseURL": "/data/gen/an/images/",
+    "width": "150px",
+    "height": "192px"
+  }
+}
 ```
 
 All fields are mandatory. `baseURL` can be a relative URL or with a FQDN. For privacy issues, you should get a copy of MPs' photos in `images/<id>/` and use a relative `baseURL`.
 
-When your dataset is ready, add this entry in the `Makefile`:
-
-```
-<id>:
-	mkdir -p json/
-	yq -c -s . data/<id>/*.yml > json/<id>.json
-	yq -c . data/<id>.yml > json/<id>-config.json
-```
-
 ## License
 
 Source code is MIT licensed. Data may have a different license.

data/an.json

+{
+  "designation": {
+    "singular": "député·e",
+    "plural": "député·es"
+  },
+  "phone": {
+    "local_prefix": "33",
+    "separator": " ",
+    "group_size": 2
+  },
+  "img": {
+    "baseURL": "/data/gen/an/images/",
+    "width": "150px",
+    "height": "192px"
+  }
+}

data/an.yml

-designation:
-  singular: "député·e"
-  plural: "député·es"
-phone:
-  local_prefix: "33"
-  separator: " "
-  group_size: 2
-img:
-  baseURL: '/images/an/'
-  width: "150px"
-  height: "192px"

data/an/abad_damien.yml

-id: PA605036
-last_name: Abad
-first_name: Damien
-group: Les Républicains
-county: Ain
-commissions:
-- "Affaires européennes"
-- "Finances"
-phone:
-- "0140636063"
-- "0474755448"
-email:
-- "contact@damien-abad.fr"
-- "damien.abad@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 605036

data/an/abadie_caroline.yml

-id: PA719866
-last_name: Abadie
-first_name: Caroline
-group: La République en Marche
-county: Isère
-commissions:
-- "Lois"
-phone:
-- "0140635047"
-email:
-- "caroline.abadie@assemblee-nationale.fr"
-twitter: AbadieCaroline
-facebook: caroline.abadie.16
-photo: 719866

data/an/acquaviva_jean-felix.yml

-id: PA719146
-last_name: Acquaviva
-first_name: Jean-Félix
-group: Libertés et Territoires
-county: Haute-Corse
-commissions:
-- "Lois"
-phone:
-- "0140630102"
-- "0495611589"
-email:
-- "jean-felix.acquaviva@assemblee-nationale.fr"
-twitter: JF_Acquaviva
-facebook: jeanfelixacquaviva
-photo: 719146

data/an/adam_damien.yml

-id: PA722038
-last_name: Adam
-first_name: Damien
-group: La République en Marche
-county: Seine-Maritime
-commissions:
-- "Affaires économiques"
-phone:
-- "0140630215"
-email:
-- "damien.adam@assemblee-nationale.fr"
-twitter: damienadam76
-facebook: damienadam76
-photo: 722038

data/an/adam_lenaick.yml

-id: PA721036
-last_name: Adam
-first_name: Lénaïck
-group: La République en Marche
-county: Guyane
-commissions:
-- "Lois"
-phone:
-- "0140637444"
-email:
-- "lenaick.adam@assemblee-nationale.fr"
-twitter: LenaickADAM
-facebook: lenaick.adam
-photo: 721036

data/an/ahamada_said.yml

-id: PA718954
-last_name: Ahamada
-first_name: Saïd
-group: La République en Marche
-county: Bouches-du-Rhône
-commissions:
-- "Finances"
-phone:
-- "0140633109"
-email:
-- "said.ahamada@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 718954

data/an/alauzet_eric.yml

-id: PA605963
-last_name: Alauzet
-first_name: Éric
-group: La République en Marche
-county: Doubs
-commissions:
-- "Finances"
-phone:
-- "0140637351"
-email:
-- "depute@alauzet.net"
-- "eric.alauzet@assemblee-nationale.fr"
-twitter: ericalauzet
-facebook: ericalauzet
-photo: 605963

data/an/ali_ramlati.yml

-id: PA724827
-last_name: Ali
-first_name: Ramlati
-group: La République en Marche
-county: Mayotte
-commissions:
-- "Affaires étrangères"
-phone:
-- "0140633103"
-email:
-- "ramlati.ali@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 724827

data/an/amadou_aude.yml

-id: PA720124
-last_name: Amadou
-first_name: Aude
-group: La République en Marche
-county: Loire-Atlantique
-commissions:
-- "Affaires étrangères"
-phone:
-- "0140630315"
-email:
-- "permanence@audeamadou.fr"
-- "aude.amadou@assemblee-nationale.fr"
-twitter: AudeAmadou
-facebook: AmadouAude
-photo: 720124

data/an/anato_patrice.yml

-id: PA721278
-last_name: Anato
-first_name: Patrice
-group: La République en Marche
-county: Seine-Saint-Denis
-commissions:
-- "Affaires européennes"
-- "Affaires économiques"
-phone:
-- "0140636930"
-email:
-- "patrice.anato@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 721278

data/an/anglade_pieyre-alexandre.yml

-id: PA721158
-last_name: Anglade
-first_name: Pieyre-Alexandre
-group: La République en Marche
-county: Français établis hors de France
-commissions:
-- "Affaires européennes"
-- "Lois"
-phone:
-- "0140636680"
-email:
-- "pieyre-alexandre.anglade@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 721158

data/an/anthoine_emmanuelle.yml

-id: PA719318
-last_name: Anthoine
-first_name: Emmanuelle
-group: Les Républicains
-county: Drôme
-commissions:
-- "Immunités"
-- "Affaires culturelles et éducation"
-phone:
-- "0140637203"
-- "0475483551"
-email:
-- "emmanuelle.anthoine@assemblee-nationale.fr"
-twitter: 
-facebook: emmanuelle.anthoine.1
-photo: 719318

data/an/ardouin_jean-philippe.yml

-id: PA719100
-last_name: Ardouin
-first_name: Jean-Philippe
-group: La République en Marche
-county: Charente-Maritime
-commissions:
-- "Défense"
-phone:
-- "0140637337"
-- "0140637336"
-- "0961670376"
-email:
-- "jean-philippe.ardouin@assemblee-nationale.fr"
-twitter: 
-facebook: jpardouin
-photo: 719100

data/an/arend_christophe.yml

-id: PA720310
-last_name: Arend
-first_name: Christophe
-group: La République en Marche
-county: Moselle
-commissions:
-- "Développement durable"
-phone:
-- "0140633210"
-email:
-- "c.arend5706@gmail.com"
-- "christophe.arend@assemblee-nationale.fr"
-twitter: 
-facebook: 
-photo: 720310