Nurture Export

Owned by George Swinnerton
Last updated: Feb 03, 2022 by Tim Whitlock (Deactivated)Version comment
2 min read

Purpose

The Nurture project has a website at https://www.nurturebiobank.org/ which includes charts hosted at https://dashboard.nurturebiobank.org , a server running Metabase.

In order to update the charts data needs to be extracted from RADAR in sqlite format and uploaded to the Metabase server.

Instructions

The intention is for this code to be part of the main RADAR project. At the moment it being merged is held up by incompatabilities when trying to apply the sqlite export destination to all existing data sources. These instructions describe running the branched code from GitHub. For more details see https://github.com/renalreg/radar/pull/7

1) Account Setup

An account called "nurture_exporrt" has been created on nww.radar.nhs.uk. The details for the account can be found in KeyPass. The user must be a member of the "radar" group to be able to access the main install's configuratoin files etc.

2) Checkout Code


[nurture_extract@nww ~]$ git clone https://github.com/renalreg/radar.git
Cloning into 'radar'...
remote: Counting objects: 33280, done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 33280 (delta 179), reused 156 (delta 95), pack-reused 33026
Receiving objects: 100% (33280/33280), 12.14 MiB | 2.63 MiB/s, done.
Resolving deltas: 100% (24458/24458), done.

3) Select Nurture Exporter Branch


[nurture_extract@nww radar]$ git checkout nurture-dashboard-exporter
Branch nurture-dashboard-exporter set up to track remote branch nurture-dashboard-exporter from origin.
Switched to a new branch 'nurture-dashboard-exporter'

4) Checkout RADAR Configs

(This is the main branch)

[nurture_extract@nww ~]$ git clone https://github.com/renalreg/radar-exports.git
Cloning into 'radar-exports'...
Username for 'https://github.com': YOUR_USERNAME
Password for 'https://YOUR_USERNAME@github.com':
remote: Counting objects: 36, done.
remote: Total 36 (delta 0), reused 0 (delta 0), pack-reused 36
Unpacking objects: 100% (36/36), done.

5) Create Python VENV


[nurture_extract@nww ~]$ virtualenv venv
New python executable in /home/nurture_extract/venv/bin/python
Installing setuptools, pip, wheel...done.

6) Activate VENV and Install RADAR (plus requirements)

source venv/bin/activate
cd radar
pip install -r requirements.txt
pip install -e .


All the prior steps have been done on the RADAR Server.

7) Run the extract script

./nurture_extract.sh

This contains:

export RADAR_SETTINGS=/etc/radar-api/settings.py

source venv/bin/activate

rm nurture-ins.db
radar-exporter --format=sqlite radar-exports/NUR-43-INS.ini nurture-ins.db
rm nurture-ckd.db
radar-exporter --format=sqlite radar-exports/NUR-43-CKD.ini nurture-ckd.db

8) Schedule SFTP to Dashboard Server


The files are transfered by a crontab job running on systems-live under an account nurture.

crontab
0 0,7,13,19 * * * /home/nurture/scripts/transfer_databases.sh


Transfer Script
scp nurture_extract@nww.radar.nhs.uk:/home/nurture_extract/databases/nurture-ckd.db ./
scp nurture_extract@nww.radar.nhs.uk:/home/nurture_extract/databases/nurture-ins.db ./
scp ./nurture-ckd.db nurture_dashboard@dashboard.nurturebiobank.org:/data/nurture_dashboard/
scp ./nurture-ins.db nurture_dashboard@dashboard.nurturebiobank.org:/data/nurture_dashboard/