File Transfers

Owned by rapolas (Unlicensed)
Last updated: Jul 23, 2019 by james.griffin (Unlicensed)Version comment
2 min read

The current development process requires testing with up to date live data on the staging server ("staging-mirth.ukrdc.nhs.uk").

In order to sync the data on the staging server with the data on the live server ("live-mirth.ukrdc.nhs.uk") a process of transferring files has been implemented. It involves running two Python scripts on the "rr-hs-test" server, which are scheduled using the Cron utility.

There are two cronjobs running on "rr-hs-test" server under pvfeed account. One is for staging - pvfeeddownloadstaging.py , another for live - pvfeeddownloadmirth.py . 

[root@rr-hs-test ~]# su - pvfeed
[pvfeed@rr-hs-test ~]$ crontab -l
*/1 * * * * flock -n .pvdownload.lck -c "scl enable python27 'python pvfeeddownloadmirth.py'" >> /var/log/pv/pvfeed.log
*/2 * * * * flock -n .pvstagingdownload.lck -c "scl enable python27 'python pvfeeddownloadstaging.py'" >> /var/log/pv/pvstagingfeed.log
[pvfeed@rr-hs-test ~]$

pvfeeddownloadmirth.py

Runs every minute. Connects to renalpatientview.org (85.118.234.56) with intersystems  username and downloads all the files from /outgoing  directory to incoming directory on rr-hs-test. After downloading a file, it removes it from the remote server.

Next it connects to sftp.ukrdc.nhs.uk (10.38.181.124)  with intersystems  username and does the same as with renalpatientview.org server.

Finally it connects to "live-mirth.ukrdc.nhs.uk" with pvfeed  username and downloads all the membership files from /data/pv/memberships/  to /memberships  directory.  The membership are then sent to staging API "https://apistaging.ukrdc.org" to register  the new memberships on the staging server.

Once it finishes with the memberships, it checks if the downloaded PV files are duplicates (same as previously received) and if not it uploads them to the "live-mirth.ukrdc.nhs.uk" server, to /data/pv/in/  directory and moves file to /incoming_staging  directory on rr-hs-test server.

pvfeeddownloadstaging.py

Runs every second minute. It simply uploads files from /incoming_staging  directory to "staging-mirth.ukrdc.nhs.uk" server to /data/pv/in/  directory, after checking if it is not a duplicate file. Once it uploads, it removes file from /incoming_staging  directory.