From 97e9d63f8c7382e9bfd7f80f98d4177bf2b86b43 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Tue, 25 Aug 2020 17:38:10 +0100 Subject: [PATCH 01/10] Update and clarify README --- README.md | 118 +++++++++++++++++++++++------------------------------- 1 file changed, 50 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index ab883ed..5747f47 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -# Pear LocalLoop Server +# LocalSpend - Server + +This repository contains the server for the LocalSpend system. ## Current Status @@ -6,35 +8,40 @@ *Development:* [![Build Status](https://travis-ci.org/Pear-Trading/Foodloop-Server.svg?branch=development)](https://travis-ci.org/Pear-Trading/Foodloop-Server) -# Testing +# Test Environment -To run the main test framework, first install all the dependencies, then run the tests: +## Running Tests + +To run the main test framework, first install all the dependencies, then run +the tests: ``` cpanm --installdeps . prove -lr ``` -To run the main framework against a PostgreSQL backend, assuming you have postgres installed, you will need some extra dependencies first: +To run the main framework against a PostgreSQL backend, assuming you have +`postgres` installed, you will need some extra dependencies first: ``` cpanm --installdeps . --with-feature postgres PEAR_TEST_PG=1 prove -lr ``` -# Minion +## Setting Up Minion -to set up minion support, you will need to create a database and user for -minion to connect to. In production his should be a PostgreSQL database, -however an SQLite db can be used in testing. +To set up Minion support, you will need to create a database and user for +it to connect to. +In production his should be a PostgreSQL database, however SQLite can be used +in testing. -To use the SQLite version, run the following commands: +To use the SQLite version, run the following command: ``` cpanm --installdeps --with-feature sqlite . ``` -And then add the following to your configuration file: +And then ensure that the following is in your configuration file: ``` minion => { @@ -42,28 +49,30 @@ And then add the following to your configuration file: }, ``` -This will then use an SQLite db for the minion backend, using `minion.db` as -the database file. To start the minion itself, run: +This will then use an SQLite DB for the Minion backend, using `minion.db` as +the database file. +To start the minion itself, run: ``` ./script/pear-local_loop minion worker ``` -# Importing Ward Data +## Importing Ward Data -To import ward data, get the ward data csv and then run the following command: +To import ward data, download CSV(s) from [here](https://www.doogal.co.uk/PostcodeDownloads.php) and then run the following command: ```shell script ./script/pear-local_loop minion job \ --enqueue 'csv_postcode_import' \ - --args '[ "/path/to/ward/csv" ]' + --args '[ "⟨ path to CSV ⟩ ]' ``` -# Setting up Entity Postcodes +## Setting Up Entity Postcodes -Assuming you have imported codepoint open, then to properly assign all - postcodes: - +1. Import Code-Point Open: + - included in `etc/` + - `./script/pear-local_loop codepoint_open --outcodes LA1` +1. Assign postcodes: ```shell script ./script/pear-local_loop minion job \ --enqueue entity_postcode_lookup @@ -81,56 +90,35 @@ psql=# alter user minion with encrypted password 'abc123'; psql=# grant all privileges on database localloop_minion to minion; ``` -# Development +# Development Environment There are a couple of setup steps to getting a development environment ready. -Use the corresponding instructions depending on what state your current setup -is in. ## First Time Setup -First, decide if you're using SQLite or PostgreSQL locally. Development supports -both, however production uses PostgreSQL. For this example we will use SQLite. -As the default config is set up for this, no configuration changes are -needed initially. So, first off, install dependencies: +1. Decide if you're using SQLite or PostgreSQL locally. + - Development supports both, however production uses PostgreSQL. + - For this example we will use SQLite. + - As the default config. is set up for this, no configuration changes are +needed initially. +1. Install dependencies: + - `cpanm --installdeps . --with-feature=sqlite --with-feature=codepoint-open` +1. Install the database: + - `./script/deploy_db install -c 'dbi:SQLite:dbname=foodloop.db'` +1. Set up the development users: + - `./script/pear-local_loop dev_data --force` + - ***DO NOT RUN ON PROD.*** +1. Start the minion: + - `./script/pear-local_loop minion worker` +1. Import ward data (see Production instructions above) +1. Set up postcodes (see Production intructions above) +1. Start the application: + - `morbo script/pear-local_loop -l http://*:3000` + - You can modify the host and port as needed. -```shell script -cpanm --installdeps . --with-feature=sqlite -``` +## Database Scripts -Then install the database: - -```shell script -./script/deploy_db install -c 'dbi:SQLite:dbname=foodloop.db' -``` - -Then set up the development users: - -```shell script -./script/pear-local_loop dev_data --force -``` - -***Note: do NOT run that script on production.*** - -Then you can start the application: - -```shell script -morbo script/pear-local_loop -l http://*:3000 -``` - -You can modify the host and port for listening as needed. - -# Old Docs - -## Local test database - -To install a local DB: - -``` -./script/deploy_db install -c 'dbi:SQLite:dbname=foodloop.db' -``` - -To do an upgrade of it after making DB changes to commit: +To upgrade the database after making changes to commit: ``` ./script/deploy_db write_ddl -c 'dbi:SQLite:dbname=foodloop.db' @@ -142,9 +130,3 @@ To redo leaderboards: ``` ./script/pear-local_loop recalc_leaderboards ``` - -To serve a test version locally of the server: - -``` -morbo script/pear-local_loop -``` From b8b841ccbf78d80a321de077a7801a1758b122f7 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Tue, 25 Aug 2020 17:44:06 +0100 Subject: [PATCH 02/10] Minor README amendments --- README.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 5747f47..7369656 100644 --- a/README.md +++ b/README.md @@ -59,13 +59,11 @@ To start the minion itself, run: ## Importing Ward Data -To import ward data, download CSV(s) from [here](https://www.doogal.co.uk/PostcodeDownloads.php) and then run the following command: +To import ward data: -```shell script -./script/pear-local_loop minion job \ - --enqueue 'csv_postcode_import' \ - --args '[ "⟨ path to CSV ⟩ ]' -``` +1. Download CSV(s) from [here](https://www.doogal.co.uk/PostcodeDownloads.php) +1. Run the following command: + -- `./script/pear-local_loop minion job --enqueue 'csv_postcode_import' --args '[ "⟨ path to CSV ⟩ ]'` ## Setting Up Entity Postcodes @@ -110,8 +108,8 @@ needed initially. - ***DO NOT RUN ON PROD.*** 1. Start the minion: - `./script/pear-local_loop minion worker` -1. Import ward data (see Production instructions above) -1. Set up postcodes (see Production intructions above) +1. Import ward data (see [instructions](#importing-ward-data) above) +1. Set up postcodes (see [intructions](#setting-up-entity-postcodes) above) 1. Start the application: - `morbo script/pear-local_loop -l http://*:3000` - You can modify the host and port as needed. From c46677bbca8795d5017fb31ec120f7ca82f52f58 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Tue, 25 Aug 2020 17:50:11 +0100 Subject: [PATCH 03/10] Minor README amendments --- README.md | 53 ++++++++++++++++++++++++----------------------------- 1 file changed, 24 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 7369656..4f641f7 100644 --- a/README.md +++ b/README.md @@ -15,18 +15,14 @@ This repository contains the server for the LocalSpend system. To run the main test framework, first install all the dependencies, then run the tests: -``` -cpanm --installdeps . -prove -lr -``` +- `cpanm --installdeps .` +- `prove -lr` To run the main framework against a PostgreSQL backend, assuming you have `postgres` installed, you will need some extra dependencies first: -``` -cpanm --installdeps . --with-feature postgres -PEAR_TEST_PG=1 prove -lr -``` +- `cpanm --installdeps . --with-feature postgres` +- `PEAR_TEST_PG=1 prove -lr` ## Setting Up Minion @@ -35,27 +31,22 @@ it to connect to. In production his should be a PostgreSQL database, however SQLite can be used in testing. -To use the SQLite version, run the following command: +To use the SQLite version: -``` -cpanm --installdeps --with-feature sqlite . -``` - -And then ensure that the following is in your configuration file: - -``` - minion => { - SQLite => 'sqlite:minion.db', - }, -``` +1. Install the SQLite dependencies: + - `cpanm --installdeps --with-feature sqlite .` +2. Ensure that the following is in your configuration file: + - ``` + minion => { + SQLite => 'sqlite:minion.db', + }, + ``` This will then use an SQLite DB for the Minion backend, using `minion.db` as the database file. -To start the minion itself, run: -``` -./script/pear-local_loop minion worker -``` +To start the minion itself, run this command: +- `./script/pear-local_loop minion worker` ## Importing Ward Data @@ -63,7 +54,11 @@ To import ward data: 1. Download CSV(s) from [here](https://www.doogal.co.uk/PostcodeDownloads.php) 1. Run the following command: - -- `./script/pear-local_loop minion job --enqueue 'csv_postcode_import' --args '[ "⟨ path to CSV ⟩ ]'` + - ```shell script + ./script/pear-local_loop minion job \ + --enqueue 'csv_postcode_import' \ + --args '[ "⟨ path to CSV ⟩ ]' + ``` ## Setting Up Entity Postcodes @@ -71,10 +66,10 @@ To import ward data: - included in `etc/` - `./script/pear-local_loop codepoint_open --outcodes LA1` 1. Assign postcodes: -```shell script -./script/pear-local_loop minion job \ - --enqueue entity_postcode_lookup -``` + - ```shell script + ./script/pear-local_loop minion job \ + --enqueue entity_postcode_lookup + ``` ## Example PostgreSQL setup From d17ba867af942ab4650a6e9a2ac8343c6dca42cf Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Tue, 25 Aug 2020 17:52:08 +0100 Subject: [PATCH 04/10] Minor README amendments --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 4f641f7..fdc3428 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,11 @@ There are a couple of setup steps to getting a development environment ready. - As the default config. is set up for this, no configuration changes are needed initially. 1. Install dependencies: - - `cpanm --installdeps . --with-feature=sqlite --with-feature=codepoint-open` + - ```shell script + cpanm --installdeps . \ + --with-feature=sqlite \ + --with-feature=codepoint-open + ``` 1. Install the database: - `./script/deploy_db install -c 'dbi:SQLite:dbname=foodloop.db'` 1. Set up the development users: @@ -113,13 +117,9 @@ needed initially. To upgrade the database after making changes to commit: -``` -./script/deploy_db write_ddl -c 'dbi:SQLite:dbname=foodloop.db' -./script/deploy_db upgrade -c 'dbi:SQLite:dbname=foodloop.db' -``` +1. `./script/deploy_db write_ddl -c 'dbi:SQLite:dbname=foodloop.db'` +1. `./script/deploy_db upgrade -c 'dbi:SQLite:dbname=foodloop.db'` To redo leaderboards: -``` -./script/pear-local_loop recalc_leaderboards -``` +1. `./script/pear-local_loop recalc_leaderboards` From 50a8139cfae3842d063d53f9bc091014046aea8d Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Tue, 25 Aug 2020 17:53:12 +0100 Subject: [PATCH 05/10] Fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fdc3428..e281e2e 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ needed initially. 1. Start the minion: - `./script/pear-local_loop minion worker` 1. Import ward data (see [instructions](#importing-ward-data) above) -1. Set up postcodes (see [intructions](#setting-up-entity-postcodes) above) +1. Set up postcodes (see [instructions](#setting-up-entity-postcodes) above) 1. Start the application: - `morbo script/pear-local_loop -l http://*:3000` - You can modify the host and port as needed. From 71de03b449239d22a2f689907cf2da92895938e3 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Wed, 26 Aug 2020 14:11:39 +0100 Subject: [PATCH 06/10] Add environment file step --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e281e2e..3ae5b1a 100644 --- a/README.md +++ b/README.md @@ -104,11 +104,12 @@ needed initially. - `./script/deploy_db install -c 'dbi:SQLite:dbname=foodloop.db'` 1. Set up the development users: - `./script/pear-local_loop dev_data --force` - - ***DO NOT RUN ON PROD.*** + - **DO NOT RUN ON PROD.** 1. Start the minion: - `./script/pear-local_loop minion worker` 1. Import ward data (see [instructions](#importing-ward-data) above) 1. Set up postcodes (see [instructions](#setting-up-entity-postcodes) above) +1. Create an `environment.dev.ts` file in `src/environments` with your API keys 1. Start the application: - `morbo script/pear-local_loop -l http://*:3000` - You can modify the host and port as needed. From 6d71d51581165f56a8c19286c89737d3e46d35f6 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Wed, 14 Oct 2020 13:40:41 +0100 Subject: [PATCH 07/10] Commit database files --- share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql | 6 +++--- share/ddl/PostgreSQL/deploy/30/001-auto.sql | 6 +++--- share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql | 6 +++--- share/ddl/SQLite/deploy/30/001-auto.sql | 6 +++--- share/ddl/_source/deploy/30/001-auto-__VERSION.yml | 2 +- share/ddl/_source/deploy/30/001-auto.yml | 2 +- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql b/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql index b8b6d70..f8a54df 100644 --- a/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql +++ b/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::PostgreSQL --- Created on Mon Sep 2 13:06:08 2019 --- +-- Created on Tue Aug 25 15:48:43 2020 +-- ; -- -- Table: dbix_class_deploymenthandler_versions diff --git a/share/ddl/PostgreSQL/deploy/30/001-auto.sql b/share/ddl/PostgreSQL/deploy/30/001-auto.sql index 357c4a8..0927410 100644 --- a/share/ddl/PostgreSQL/deploy/30/001-auto.sql +++ b/share/ddl/PostgreSQL/deploy/30/001-auto.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::PostgreSQL --- Created on Mon Sep 2 13:06:08 2019 --- +-- Created on Tue Aug 25 15:48:42 2020 +-- ; -- -- Table: account_tokens diff --git a/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql b/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql index 21f87c4..d08c60c 100644 --- a/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql +++ b/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::SQLite --- Created on Mon Sep 2 13:06:08 2019 --- +-- Created on Tue Aug 25 15:48:43 2020 +-- ; BEGIN TRANSACTION; diff --git a/share/ddl/SQLite/deploy/30/001-auto.sql b/share/ddl/SQLite/deploy/30/001-auto.sql index 320ab24..2962bd1 100644 --- a/share/ddl/SQLite/deploy/30/001-auto.sql +++ b/share/ddl/SQLite/deploy/30/001-auto.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::SQLite --- Created on Mon Sep 2 13:06:08 2019 --- +-- Created on Tue Aug 25 15:48:43 2020 +-- ; BEGIN TRANSACTION; diff --git a/share/ddl/_source/deploy/30/001-auto-__VERSION.yml b/share/ddl/_source/deploy/30/001-auto-__VERSION.yml index 2cc0cbf..7f96b5c 100644 --- a/share/ddl/_source/deploy/30/001-auto-__VERSION.yml +++ b/share/ddl/_source/deploy/30/001-auto-__VERSION.yml @@ -88,4 +88,4 @@ translator: producer_type: SQL::Translator::Producer::YAML show_warnings: 0 trace: 0 - version: 0.11024 + version: 1.61 diff --git a/share/ddl/_source/deploy/30/001-auto.yml b/share/ddl/_source/deploy/30/001-auto.yml index 7e4a6d2..0c2e4c7 100644 --- a/share/ddl/_source/deploy/30/001-auto.yml +++ b/share/ddl/_source/deploy/30/001-auto.yml @@ -3547,4 +3547,4 @@ translator: producer_type: SQL::Translator::Producer::YAML show_warnings: 0 trace: 0 - version: 0.11024 + version: 1.61 From 4a71b7cf129686bbf2c77ca0ea716da8d78e33c8 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Wed, 14 Oct 2020 13:40:56 +0100 Subject: [PATCH 08/10] Add troubleshooting information --- README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/README.md b/README.md index 3ae5b1a..abc24f6 100644 --- a/README.md +++ b/README.md @@ -124,3 +124,19 @@ To upgrade the database after making changes to commit: To redo leaderboards: 1. `./script/pear-local_loop recalc_leaderboards` + +# Troubleshooting + +## ‘Can't write to /usr/local/share/perl/5.30.0 and /usr/local/bin: Installing modules to /home//perl5’ when running `cpanm` commands + +Intall `local::lib` by running the following commands: + +``` + cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib) +``` + +NB: You must run this in each Terminal window; I don't know why. + +## ‘Can't load application from file "/script/pear-local_loop": Can't locate Data/UUID.pm in @INC (you may need to install the DATA::UUID module)’ when running server + +Ensure you have run the `cpan --installdeps` command. From a3923b7f6f2b26968eb6abadffef6bc9d53d5172 Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Sun, 18 Oct 2020 14:38:47 +0100 Subject: [PATCH 09/10] Revert "Commit database files" This reverts commit c5878b3a08e3b77013f5ac6ff3d5c2746767d6ef. --- share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql | 6 +++--- share/ddl/PostgreSQL/deploy/30/001-auto.sql | 6 +++--- share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql | 6 +++--- share/ddl/SQLite/deploy/30/001-auto.sql | 6 +++--- share/ddl/_source/deploy/30/001-auto-__VERSION.yml | 2 +- share/ddl/_source/deploy/30/001-auto.yml | 2 +- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql b/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql index f8a54df..b8b6d70 100644 --- a/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql +++ b/share/ddl/PostgreSQL/deploy/30/001-auto-__VERSION.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::PostgreSQL --- Created on Tue Aug 25 15:48:43 2020 --- +-- Created on Mon Sep 2 13:06:08 2019 +-- ; -- -- Table: dbix_class_deploymenthandler_versions diff --git a/share/ddl/PostgreSQL/deploy/30/001-auto.sql b/share/ddl/PostgreSQL/deploy/30/001-auto.sql index 0927410..357c4a8 100644 --- a/share/ddl/PostgreSQL/deploy/30/001-auto.sql +++ b/share/ddl/PostgreSQL/deploy/30/001-auto.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::PostgreSQL --- Created on Tue Aug 25 15:48:42 2020 --- +-- Created on Mon Sep 2 13:06:08 2019 +-- ; -- -- Table: account_tokens diff --git a/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql b/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql index d08c60c..21f87c4 100644 --- a/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql +++ b/share/ddl/SQLite/deploy/30/001-auto-__VERSION.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::SQLite --- Created on Tue Aug 25 15:48:43 2020 --- +-- Created on Mon Sep 2 13:06:08 2019 +-- ; BEGIN TRANSACTION; diff --git a/share/ddl/SQLite/deploy/30/001-auto.sql b/share/ddl/SQLite/deploy/30/001-auto.sql index 2962bd1..320ab24 100644 --- a/share/ddl/SQLite/deploy/30/001-auto.sql +++ b/share/ddl/SQLite/deploy/30/001-auto.sql @@ -1,7 +1,7 @@ --- +-- -- Created by SQL::Translator::Producer::SQLite --- Created on Tue Aug 25 15:48:43 2020 --- +-- Created on Mon Sep 2 13:06:08 2019 +-- ; BEGIN TRANSACTION; diff --git a/share/ddl/_source/deploy/30/001-auto-__VERSION.yml b/share/ddl/_source/deploy/30/001-auto-__VERSION.yml index 7f96b5c..2cc0cbf 100644 --- a/share/ddl/_source/deploy/30/001-auto-__VERSION.yml +++ b/share/ddl/_source/deploy/30/001-auto-__VERSION.yml @@ -88,4 +88,4 @@ translator: producer_type: SQL::Translator::Producer::YAML show_warnings: 0 trace: 0 - version: 1.61 + version: 0.11024 diff --git a/share/ddl/_source/deploy/30/001-auto.yml b/share/ddl/_source/deploy/30/001-auto.yml index 0c2e4c7..7e4a6d2 100644 --- a/share/ddl/_source/deploy/30/001-auto.yml +++ b/share/ddl/_source/deploy/30/001-auto.yml @@ -3547,4 +3547,4 @@ translator: producer_type: SQL::Translator::Producer::YAML show_warnings: 0 trace: 0 - version: 1.61 + version: 0.11024 From 350113807473d32c2baae475cda2ae1397aa941a Mon Sep 17 00:00:00 2001 From: Ben Goldsworthy Date: Wed, 18 Nov 2020 23:01:18 +0000 Subject: [PATCH 10/10] Add db upgrade step --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index abc24f6..a6d9c6e 100644 --- a/README.md +++ b/README.md @@ -118,6 +118,7 @@ needed initially. To upgrade the database after making changes to commit: +1. Increment the `$VERSION` number in `lib/Pear/LocalLoop/Schema.pm` 1. `./script/deploy_db write_ddl -c 'dbi:SQLite:dbname=foodloop.db'` 1. `./script/deploy_db upgrade -c 'dbi:SQLite:dbname=foodloop.db'`