Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRAFT: Brownfield land bulk load #132

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

DRAFT: Brownfield land bulk load #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ce37bfa
Add a small sample file
mdsmith-scottlogic Sep 4, 2023
9f2d52a
Add a sample with three endpoints
mdsmith-scottlogic Sep 5, 2023
e5860ed
typo
mdsmith-scottlogic Sep 6, 2023
be612ed
Add test for duplicate row.
mdsmith-scottlogic Sep 6, 2023
ae66b5b
Added test data and simple test script
mdsmith-scottlogic Sep 8, 2023
e5da816
Add an exerpt of rowsd not yet
mdsmith-scottlogic Sep 8, 2023
cdc8685
Remove extraneous suffix
mdsmith-scottlogic Sep 8, 2023
1ef19db
Updated test excerpt
mdsmith-scottlogic Sep 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Development requires Python 3.6.2 or later, we recommend using a [virtual enviro

make init
make
python -m digital-land --help
python -m digital_land --help


## Commands Guide
Expand Down
340 changes: 340 additions & 0 deletions tests/data/brownfield_land/brownfield.csv
View file
Open in desktop

Large diffs are not rendered by default.

52 changes: 52 additions & 0 deletions tests/data/brownfield_land/brownfield_excerpt_for_testing.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
organisation,pipelines,documentation-url,endpoint-url,start-date,endpoint added,endpoint ready to be added,old endpoint need stopping?,Checked with validator?
local-authority-eng:BAS,brownfield-land,https://beta.bathnes.gov.uk/policy-and-documents-library/brownfield-register,https://beta.bathnes.gov.uk/sites/default/files/2022-12/Brownfield_register%202022.csv,21/12/2022,False,Yes,Yes,FALSE
local-authority-eng:BDF,brownfield-land,https://www.bedford.gov.uk/planning-and-building-control/planning-policy/brownfield-land-register,https://www.bedford.gov.uk/media/5315/download?inline,31/10/2022,False,Yes,Yes,FALSE
local-authority-eng:BDG,brownfield-land,https://www.lbbd.gov.uk/planning-building-control-and-local-land-charges/planning-guidance-and-policies/brownfield-land,https://www.lbbd.gov.uk/sites/default/files/2022-09/Barking-and-Dagenham-Brownfield-Land-Register%20%283%29.xlsx,31/12/2017,False,Yes,Yes,FALSE
local-authority-eng:CLD,brownfield-land,https://dataworks.calderdale.gov.uk/dataset/vddg5/brownfield-land-register,https://dataworks.calderdale.gov.uk/download/vddg5/l0w/Brownfield%20Land%20Register%202022.csv,21/12/2022,False,Yes,Yes,FALSE
local-authority-eng:DBY,brownfield-land,Brownfield Land Register - Derbyshire Dales District Council,https://www.derbyshiredales.gov.uk/documents/the-brownfield-land-register-2022-csv-file/download,30/12/2022,False,Yes,Yes,FALSE
local-authority-eng:DUD,brownfield-land,Brownfield Land Register | Dudley Council,https://www.dudley.gov.uk/media/14362/brownfield-land.csv,30/12/2022,False,Yes,Yes,FALSE
local-authority-eng:ENF,brownfield-land,https://www.enfield.gov.uk/services/planning/registers,https://www.enfield.gov.uk/__data/assets/excel_doc/0018/4608/planning-policy-information-enfield-blr-part-1.xlsx,31/12/2017,False,Yes,Yes,FALSE
local-authority-eng:FOE,brownfield-land,https://www.fdean.gov.uk/planning-and-building/planning-policy/supporting-information-for-the-local-plan-2041/monitoring-and-land-reports/brownfield-land-register/,https://www.fdean.gov.uk/media/h4jj44cj/brownfield-land-register-for-2022.xlsx,12/1/2022,False,Yes,Yes,FALSE
local-authority-eng:FYL,brownfield-land,https://new.fylde.gov.uk/resident/planning/planning-policy-local-plan/brownfield-register/,https://new.fylde.gov.uk/wp-content/uploads/2023/06/Fylde-Council-Brownfield-Land-Register-April-2023.pdf,28/04/2023,False,Yes,Yes,FALSE
local-authority-eng:GAT,brownfield-land,https://www.gateshead.gov.uk/article/3017/Brownfield-Register-April-2022,https://www.gateshead.gov.uk/media/32763/Gateshead-Brownfield-Register/excel/gateshead_brownfieldregister_2022-04-07_rev1.csv,07/04/2022,False,Yes,Yes,FALSE
local-authority-eng:GLA,brownfield-land,https://data.london.gov.uk/dataset/brownfield_register,https://data.london.gov.uk/download/brownfield_register/7ef14d04-8347-4133-b8a7-7c72415f25f1/Brownfield%20Register%20data%20csv.csv,12/07/2020,False,Yes,Yes,FALSE
local-authority-eng:GLS,brownfield-land,Brownfield Land Register - Gloucester City Council,https://www.gloucester.gov.uk/media/8454/brownfield-land-register-dec22.csv,30/12/2022,False,Yes,Yes,FALSE
local-authority-eng:GMCA,brownfield-land,,https://www.manchester.gov.uk/download/downloads/id/29070/brownfield_land_register_2022.csv,,False,Yes,Yes,FALSE
local-authority-eng:GOS,brownfield-land,https://www.gosport.gov.uk/brownfield-land-register,https://www.gosport.gov.uk/article/1777/Brownfield-Land-Register-CSV-redirect,20/12/2022,False,Yes,Yes,FALSE
local-authority-eng:HAL,brownfield-land,https://www.data.gov.uk/dataset/7c787a99-37a7-495f-8ae8-b3f3e69a8890/halton-brownfield-land-register,https://councillors.halton.gov.uk/documents/s49555/DCC%20Misc%20Item%20-%20draft%20Brownfield%20Register.pdf ,12/09/2017,False,Yes,Yes,FALSE
local-authority-eng:HAM,brownfield-land,https://www.easthants.gov.uk/planning-services/planning-policy/brownfield-land-register,https://www.easthants.gov.uk/media/6377/download?inline,,False,Yes,Yes,FALSE
local-authority-eng:HAS,brownfield-land,https://www.hastings.gov.uk/planning/policy/BrownfieldRegister/,https://www.hastings.gov.uk/content/planning/planning_policy/pdfs/brownfield_land_register.csv,10/12/2021,False,Yes,Yes,FALSE
local-authority-eng:HCK,brownfield-land,https://hackney.gov.uk/brownfield-land-register,https://drive.google.com/file/d/1Zqx9qJVjEtIyb2mCETkiWDzf7XDfxHzu/view?usp=share_link,31/12/2017,False,Yes,Yes,FALSE
local-authority-eng:HEF,brownfield-land,https://www.herefordshire.gov.uk/downloads/download/1832/brownfield_sites_register,https://www.herefordshire.gov.uk/downloads/file/13144/brownfield-sites-register-2021-csv,20/12/2021,False,Yes,Yes,FALSE
local-authority-eng:HIL,brownfield-land,https://www.hillingdon.gov.uk/brownfield-register,https://www.hillingdon.gov.uk/media/3054/Hillingdon-brownfield-land-register-CSV-file/excel/6m2020_Brownfield_Land_Register_2.csv,31/05/2020,False,Yes,Yes,FALSE
local-authority-eng:HRT,brownfield-land,"east herts - https://www.eastherts.gov.uk/planning-building/planning-policy/brownfield-land-register , north herts - https://www.north-herts.gov.uk/brownfield-land-register","east herts - https://www.eastherts.gov.uk/sites/default/files/documents/Brownfield_Land_Register-_Excel.xlsx , north herts - https://www.north-herts.gov.uk/sites/default/files/2022-12/Brownfield%20Land%20Register%202022%20-%20Website%20Copy.xlsx",,False,Yes,Yes,FALSE
local-authority-eng:HRW,brownfield-land,https://www.harrow.gov.uk/planning-developments/brownfield-land-register,https://data.london.gov.uk/download/brownfield_register/7ef14d04-8347-4133-b8a7-7c72415f25f1/Brownfield%20Register%20data%20csv.csv,,False,Yes,Yes,FALSE
local-authority-eng:HRY,brownfield-land,https://www.haringey.gov.uk/planning-and-building-control/planning/planning-policy/localism/brownfield-register,https://www.haringey.gov.uk/sites/haringeygovuk/files/brownfield_register.csv,02/02/2021,False,Yes,Yes,FALSE
local-authority-eng:HYN,brownfield-land,https://www.hyndburnbc.gov.uk/hyndburn-brownfield-register-2/,https://www.hyndburnbc.gov.uk/download-package/brownfield-register/?wpdmdl=5825&refresh=64aeb537017211689171255&ind=1513269338843&filename=HBC%202017%20Part%201%20Brownfield%20Register.csv,06/12/2017,False,Yes,Yes,FALSE
local-authority-eng:IPS,brownfield-land,https://www.ipswich.gov.uk/brownfieldlandregister,https://www.ipswich.gov.uk/sites/default/files/brownfieldlandregisteripswich.csv,15/12/2022,False,Yes,Yes,FALSE
local-authority-eng:KEC,brownfield-land,"https://www.rbkc.gov.uk/planning-and-building-control/planning-policy/brownfield-land-register#:~:text=The%20Council's%20Brownfield%20Land%20Register,that%20meet%20the%20relevant%20criteria%20.",https://www.rbkc.gov.uk/media/document/brownfield-land-register-2022,31/12/2022,False,Yes,Yes,FALSE
local-authority-eng:KTT,brownfield-land,"https://www.kingston.gov.uk/policy/brownfield-land-register-1#:~:text=The%20council%20is%20legally%20required,agricultural%20use%20or%20residential%20gardens.",https://www.kingston.gov.uk/downloads/file/94/kingston-brownfield-register-december-2017,12/12/2017,False,Yes,Yes,FALSE
local-authority-eng:LAN,brownfield-land,https://www.westlancs.gov.uk/planning/planning-policy/brownfield-land-register.aspx,https://www.westlancs.gov.uk/media/546541/blr-2021_west-lancs.csv,,False,Yes,Yes,FALSE
local-authority-eng:LBH,brownfield-land,https://www.lambeth.gov.uk/planning-building-control/planning-policy-guidance/brownfield-land-register,https://www.lambeth.gov.uk/sites/default/files/2022-12/brownfield-land.csv,15/12/2022,False,Yes,Yes,FALSE
local-authority-eng:LEC,brownfield-land,https://www.nwleics.gov.uk/pages/brownfield_land_register,https://www.nwleics.gov.uk/files/documents/brownfield_land_register_2022/northwestleicestershire_brownfieldregister_2022-12-31_rev1.csv,31/12/2022,False,Yes,Yes,FALSE
local-authority-eng:LEW,brownfield-land,https://lewisham.gov.uk/myservices/planning/policy/brownfield-land-register,https://lewisham.gov.uk/-/media/lewisham-brownfield-land-register-csv-dec-2022.ashx,25/10/2022,False,Yes,Yes,FALSE
local-authority-eng:LIV,brownfield-land,https://liverpool.gov.uk/planning-and-building-control/plan-making-in-liverpool/evidence-monitoring-and-information/brownfield-land-register/,https://liverpool.gov.uk/media/ajxbwh5v/brownfield-land-register-2021.xlsx,31/12/2021,False,Yes,Yes,FALSE
local-authority-eng:MAI,brownfield-land,https://maidstone.gov.uk/home/privacy-and-cookies/service-level-privacy-notices/planning,https://maidstone.gov.uk/__data/assets/pdf_file/0006/276180/Brownfield-Land-Register-Part-1-May-2019.pdf,19/05/2019,False,Yes,Yes,FALSE
local-authority-eng:MDB,brownfield-land,Brownfield land register | Middlesbrough Council,https://www.middlesbrough.gov.uk/media/tb5fa3mn/brownfield-sites-december-2022-update.csv,20/12/2022,False,Yes,Yes,FALSE
local-authority-eng:MOL,brownfield-land,Brownfield Land Register | Mole Valley District Council,https://www.molevalley.gov.uk/sites/default/files/home/building-planning/local-plans/brownfield-land_0.csv,07/10/2022,False,Yes,Yes,FALSE
local-authority-eng:NFK,brownfield-land,https://www.southnorfolkandbroadland.gov.uk/future-development/brownfield-register-1,https://www.southnorfolkandbroadland.gov.uk/downloads/file/4091/south-norfolk-council-brownfield-register-2022-csv,,False,Yes,Yes,FALSE
local-authority-eng:NTT,brownfield-land,https://www.opendatanottingham.org.uk/dataset.aspx?id=139,"part 1 - https://geoserver.nottinghamcity.gov.uk/opendata/csv/spatial/ncc_Brownfield_Register.csv?t=638290805834593883 , part 2 - https://geoserver.nottinghamcity.gov.uk/opendata/csv/spatial/ncc_Brownfield_Register_Part2.csv?t=638290805834593883",,False,Yes,Yes,FALSE
local-authority-eng:NTY,brownfield-land,Brownfield Land Register | North Tyneside Council,https://my.northtyneside.gov.uk/sites/default/files/web-page-related-files/brownfield-land_1.csv,15/12/2022,False,Yes,Yes,FALSE
local-authority-eng:NWM,brownfield-land,https://www.newham.gov.uk/homepage/177/brownfield-register,https://www.newham.gov.uk/downloads/file/5462/newhambrownfieldregister-dec-2022,06/12/2022,False,Yes,Yes,FALSE
local-authority-eng:OLD,brownfield-land,https://www.oldham.gov.uk/homepage/1386/brownfield_land_register,https://www.oldham.gov.uk/download/downloads/id/7569/oldham_brownfield_register_2022_csv.xlsx,26/02/2021,False,Yes,Yes,FALSE
local-authority-eng:PRE,brownfield-land,https://www.preston.gov.uk/article/1196/Brownfield-Land-Register,https://www.preston.gov.uk/media/1736/Brownfield-Register/pdf/Brownfield-Land-Register-November-2018_6_1.pdf,19/11/2018,False,Yes,Yes,FALSE
local-authority-eng:PTE,brownfield-land,https://www.peterborough.gov.uk/council/planning-and-development/planning-policies/brownfield-site-register,https://cccandpcc.sharepoint.com/:x:/r/sites/PCCPlanningPolicyPublicData/_layouts/15/Doc.aspx?sourcedoc=%7B19A8CF3B-4050-438C-B8E1-3BBF70C21C61%7D&file=Brownfield%20Register%202022.csv,20/12/2022,False,Yes,Yes,FALSE
local-authority-eng:RCC,brownfield-land,https://www.redcar-cleveland.gov.uk/planning/local-plan/local-plan-documents-and-evidence/five-year-housing-land-supply-brownfield-land,https://www.redcar-cleveland.gov.uk/sites/default/files/2022-10/Brownfield%20Land%20Register.xlsx,24/08/2022,False,Yes,Yes,FALSE
local-authority-eng:SFK,brownfield-land,"east - https://www.eastsuffolk.gov.uk/planning/brownfield-land-register/ , west - https://www.westsuffolk.gov.uk/planning/Planning_Policies/brownfield-register.cfm","east - https://www.eastsuffolk.gov.uk/assets/Planning/Brownfield-Land-Register/brownfield-land.csv , west - https://www.westsuffolk.gov.uk/planning/Planning_Policies/upload/Brownfield-Register-2022-final.csv",,False,Yes,Yes,FALSE
local-authority-eng:STF,brownfield-land,Brownfield Land Register - Staffordshire Moorlands District Council (staffsmoorlands.gov.uk),https://www.staffsmoorlands.gov.uk/media/4895/brownfield-land.csv-2022/Excel/sm_brownfield_land_22_csv.csv?m=1671180323880,13/12/2022,False,Yes,Yes,Errors
local-authority-eng:STN,brownfield-land,Brownfield Register - London Datastore,https://data.london.gov.uk/download/brownfield_register/7ef14d04-8347-4133-b8a7-7c72415f25f1/Brownfield%20Register%20data%20csv.csv,12/7/2020,False,Yes,Yes,Yes
local-authority-eng:SUF,brownfield-land,,,,False,Yes,Yes,FALSE
local-authority-eng:SWL,brownfield-land,Local Plans - Brownfield land register (swale.gov.uk),https://swale.gov.uk/__data/assets/file/0007/435787/Brownfield-Land-Register.csv,21/12/2022,False,Yes,Yes,FALSE
local-authority-eng:TEI,brownfield-land,https://www.teignbridge.gov.uk/planning/local-plans-and-policy/brownfield-register/,https://www.teignbridge.gov.uk/media/mt2pw4qe/2022-bfr.xlsx,23/11/2022,False,Yes,Yes,FALSE
local-authority-eng:TUN,brownfield-land,Brownfield Register for Tunbridge Wells borough,https://tunbridgewells.gov.uk/__data/assets/file/0008/437858/Brownfield-Register-Part-1-2022-2023.csv,09/01/2023,False,Yes,Yes,FALSE
local-authority-eng:WEY,brownfield-land,Brownfield land register - Dorset Council,https://www.dorsetcouncil.gov.uk/documents/35024/288820/Brownfield+land.csv/d2447101-61fd-11cb-c014-79aa041e4c42?t=1625743914942,04/01/2020,False,Yes,Yes,FALSE
2 changes: 2 additions & 0 deletions tests/data/brownfield_land/small_sample_1.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
pipelines,organisation,documentation-url,endpoint-url,start-date
brownfield-land,local-authority-eng:DUD,https://www.dudley.gov.uk/resident/planning/brownfield-land-register/,https://www.dudley.gov.uk/media/14362/brownfield-land.csv,30/12/2022
4 changes: 4 additions & 0 deletions tests/data/brownfield_land/small_sample_2.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
pipelines,organisation,documentation-url,endpoint-url,start-date
brownfield-land,local-authority-eng:DUD,https://www.dudley.gov.uk/resident/planning/brownfield-land-register/,https://www.dudley.gov.uk/media/14362/brownfield-land.csv,30/12/2022
brownfield-land,local-authority-eng:LEC,https://www.nwleics.gov.uk/pages/brownfield_land_register,https://www.nwleics.gov.uk/files/documents/brownfield_land_register_2022/northwestleicestershire_brownfieldregister_2022-12-31_rev1.csv,31/12/2022
brownfield-land,local-authority-eng:RCC,https://www.redcar-cleveland.gov.uk/planning/local-plan/local-plan-documents-and-evidence/five-year-housing-land-supply-brownfield-land,https://www.redcar-cleveland.gov.uk/sites/default/files/2022-10/Brownfield%20Land%20Register.xlsx,24/08/2022
5 changes: 5 additions & 0 deletions tests/data/brownfield_land/small_sample_2_duplicagte_row.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pipelines,organisation,documentation-url,endpoint-url,start-date
brownfield-land,local-authority-eng:DUD,https://www.dudley.gov.uk/resident/planning/brownfield-land-register/,https://www.dudley.gov.uk/media/14362/brownfield-land.csv,30/12/2022
brownfield-land,local-authority-eng:LEC,https://www.nwleics.gov.uk/pages/brownfield_land_register,https://www.nwleics.gov.uk/files/documents/brownfield_land_register_2022/northwestleicestershire_brownfieldregister_2022-12-31_rev1.csv,31/12/2022
brownfield-land,local-authority-eng:RCC,https://www.redcar-cleveland.gov.uk/planning/local-plan/local-plan-documents-and-evidence/five-year-housing-land-supply-brownfield-land,https://www.redcar-cleveland.gov.uk/sites/default/files/2022-10/Brownfield%20Land%20Register.xlsx,24/08/2022
brownfield-land,local-authority-eng:DUD,https://www.dudley.gov.uk/resident/planning/brownfield-land-register/,https://www.dudley.gov.uk/media/14362/brownfield-land.csv,30/12/2022
8 changes: 8 additions & 0 deletions tests/data/brownfield_land/test1.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash
echo "Welcome to DPD collection-add-source"

digital-land collection-add-source brownfield-land https://www.dudley.gov.uk/media/14362/brownfield-land.csv organisation local-authority-eng:DUD documentation-url https://www.dudley.gov.uk/resident/planning/brownfield-land-register/ start-date 2023-9-12

digital-land collection-add-source brownfield-land https://www.nwleics.gov.uk/files/documents/brownfield_land_register_2022/northwestleicestershire_brownfieldregister_2022-12-31_rev1.csv organisation local-authority-eng:LEC documentation-url https://www.nwleics.gov.uk/pages/brownfield_land_register

digital-land collection-add-source brownfield-land https://www.redcar-cleveland.gov.uk/sites/default/files/2022-10/Brownfield%20Land%20Register.xlsx organisation local-authority-eng:RCC documentation-url https://www.redcar-cleveland.gov.uk/planning/local-plan/local-plan-documents-and-evidence/five-year-housing-land-supply-brownfield-land