George Swinnerton has not added much to the UKRR Extract Specification in the last week. He will add details of what the minimum data is required to make a minimum extract that can be validated. This will identify the minimum code that needs to be developed to create an extract. This will be the initial software goal when there is sufficient data in the Kings file to allow the development to be done.
George Swinnerton historic surveys have been imported into the live system resulting in around 150 errors. of these 44 were caused by one or more surveys not being completed, ~8 had missing information and the rest George Swinnerton has corrected. The loading identified a number of issues with coding that should have been fully ironed out on staging before loading on live. To ensure the 2018 load goes smoothly they will be loaded on staging first. This will also provide an idea of the errors to expect for the the GUI to correct and indicate the numbers. Some could be sorted prior to loading to live during testing of the Error GUI.
rapolas (Unlicensed) has deployed the deduplicating code that prevents the same file being repeatedly loaded. Since Tuesday 77% of the files received have been duplicates. 3 or 4 units make up the majority of the offenders. Top ten offenders listed below
Count Unit 59243 RAZ 30112 ree01 26990 RQR00 11661 RM301 9821 RM574 9443 RF201 7268 RRK02 6959 RAE05 6872 RCB55 6768 RTD01 Before deploying to live we need to verify no data is being missed due to this. In addition we need to ensure Units are aware of the change so they don't raise queries because they see no file imported when they know one was sent but has been blocked due to de-duplication. Initially Rapolas will compare some of the more frequently blocked patients on staging and live and confirm that the same data is on both staging and live PV. Once this is done we can ask Shaun to do a more detailed check then deploy to live if all is well.
One site REJ01 has produced some interesting exceptions "duplicate key already exists" This appears to be because they send two "identical" files except the first has "pending" for the results and the second has the actual results. This results in two files that are not identical but the second should overwrite the first. Due to the way they are uploaded and processed they tend to process at the same time so there can be a race condition for creating the result lines hence they can both end up trying to create the same result line during processing and one fails. This is caused by a combination of factors but fundamentally because they send a result file with pending in.
George Swinnerton hasn't done the comparison of the PV Memberships. He has updated the code to generate proper error if a PV file is received with no PV membership in place. This needs testing and deploying to staging for proper testing. When this code is in place on live we will need to ensure there is a way units are kept informed about files being sent without the PV account being set up.
George Swinnerton has told kings about the errors in their RDA files and suggested how some of the data should look based on data we have in the renalreg database. He has not had any response. In addition he has proposed a visit to discuss the extract.
rapolas (Unlicensed) has created some documentation on how to create a local test environment for the UKRDC on your own desktop. George Swinnerton plans to do this so he can do more comprehensive testing locally.
Tim Whitlock (Deactivated) has configured the automatic upload of the Kings RDA files to the staging server. This currently runs at 2.00 am each night as kings upload their files currently at 1.00am. These can be changed when more interactive testing is required. The live feed and replicating it to the staging system for one or all sites was discussed. The suggestion is to modify PV-DECRYPT to read a config file that identifies all the units and whether the file should be copied to the rdastaging or rdadev download folders as well as live. This will allow each system to manage its own downloads and should reduce some of the issues we have seen with a single downloader/distribution process as used with PV files.
A meeting to discuss the validation tasks has not occurred yet.
Simplified
George Swinnerton the basic code is in place it just needs to have the filters added to remove data items not included in the DSA. This is due on Thursday Once Simplified are satisfied a daily feed will need to be configured. The code need access to PV DB and UKRDC DB so will need to run on a server in Bristol.
Action items
George Swinnerton to complete simplified extract of data example for supply by once approved a daily feed will need configuring.