python-ags4 1.2.0 | pandas 2.3.3 === Step 1: the A417 Missing Link in the BGS AGS borehole index === HE551505 A417 MISSING LINK GROUND INVESTIGATION 112 exploratory holes | deepest 105.0 m | E 392,081 to 394,938 | N 213,164 to 216,419 bgs_loca_id loca_id x y loca_fdep 2026081011400291004 CP102 392080.9 215724.9 20.00 2026081011400291005 CP104 392524.7 215642.0 15.00 2026081011400291006 CP104A 392525.2 215641.5 16.76 === Step 2: download as AGS4, 40 holes per request === batch01_202608101140029.ags: 476 kB, 3,958 lines batch02_202608101140029.ags: 493 kB, 3,806 lines batch03_202608101140029.ags: 112 kB, 1,102 lines === Step 3: what an AGS4 file looks like === "GROUP","LOCA" "HEADING","LOCA_ID","LOCA_TYPE","LOCA_STAT","LOCA_NATE","LOCA_NATN","LOCA_GREF","LOCA_GL","LOCA_ ... "UNIT","","","","m","m","","m","","m","yyyy-mm-dd","","","yyyy-mm-dd","","m","m","m","","","m"," ... "TYPE","ID","PA","PA","U","U","X","U","X","U","DT","X","X","DT","X","X","X","X","X","X","X","X", ... "DATA","CP102","RC","FINAL","392080.9","215724.9","","127.55","","20","","","","","","","","","" ... "DATA","CP104","CP","FINAL","392524.7","215642","","148","Borehole terminated at 15.00m followin ... === Step 4: read every group into pandas === 16 groups in batch01_202608101140029.ags: PROJ UNIT TYPE TRAN FILE DICT ABBR LOCA GEOL BKFL CDIA PIPE WSTG WSTD ISPT SAMP HEADING LOCA_ID LOCA_TYPE LOCA_NATE LOCA_NATN LOCA_GL UNIT m m m TYPE ID PA U U U DATA CP102 RC 392080.9 215724.9 127.55 DATA CP104 CP 392524.7 215642 148 All three files stacked, DATA rows per group: SAMP 4,555 Sample Information GEOL 1,564 Field Geological Descriptions ISPT 415 Standard Penetration Test Results BKFL 354 Exploratory Hole Backfill Details PIPE 136 Monitoring Installation Pipe Work CDIA 127 Casing Diameter by Depth LOCA 112 Location Details WSTG 24 Water Strike - General WSTD 23 Water Strike - Details === Step 5: LOCA, GEOL and SAMP === LOCA_TYPE: {'RC': 71, 'TP': 30, 'CPT': 6, 'RO': 3, 'CP': 2} LOCA_CLST: {'PHASE 2A.1': 99, 'PHASE 1': 8, 'PHASE 2A': 5} ground level 127.5 to 294.1 m OD | final depth 0.8 to 105.0 m GEOL: 1,564 strata in 112 holes. One borehole, top to bottom: GEOL_TOP GEOL_BASE GEOL_GEOL GEOL_BGS GEOL_DESC 0.00 0.20 TOP Grass over soft dark brown gravelly silty CLAY ... 0.20 0.75 HDD HEAD Soft to firm orangish brown slightly gravelly C... 0.75 1.20 HDD HEAD Firm orange and light grey slightly sandy sligh... 1.20 2.60 FEF FE Stiff bluish grey mottled orangish brown and re... 2.60 2.70 IOG SALS Limited recovery. Single angular irregular lime... 2.70 3.30 No recovery. 3.30 4.40 IOG SALS Weak to medium strong bioclastic LIMESTONE with... 4.40 6.75 IOG SALS Weak light brownish grey and white bioclastic L... 6.75 10.50 IOG SALS Weak light brown bioclastic LIMESTONE with rare... 10.50 11.60 IOG SALS Extremely weak light greyish brown bioclastic L... 11.60 12.95 IOG SALS Weak light brown bioclastic LIMESTONE. Fracture... 12.95 13.65 IOG SALS Weak light greyish brown bioclastic and ooidal ... 13.65 16.90 IOG SALS Medium strong light brown and light grey biocla... 16.90 17.35 IOG SALS Extremely weak light grey bioclastic LIMESTONE,... 17.35 19.60 IOG ASLS Weak light grey and light brown bioclastic LIME... 19.60 22.20 IOG BLPL Weak light brown bioclastic and ooidal LIMESTON... SAMP: 4,555 samples. SAMP_TYPE: {'C': 1855, 'CS': 838, 'D': 830, 'ES': 370, 'B': 326, 'L': 205, 'EW': 78, 'UT': 26} C = Core sample | CS = Core sample (prepared) | D = Small disturbed sample | ES = Soil sample for environmental testing | B = Bulk disturbed sample | L = Liner sample (dynamic) | EW = Water sample for environmental testing | UT = Thin wall open drive tube sampler === Step 6: validate the downloads with the AGS checker === batch01_202608101140029.ags batch02_202608101140029.ags batch03_202608101140029.ags total AGS Format Rule 2a 3958 3806 1102 8866 AGS Format Rule 19a 8 8 5 21 AGS Format Rule 19b 6 6 6 18 AGS Format Rule 1 1 3 0 4 AGS Format Rule 20 2 2 2 6 AGS Format Rule 7 5 5 4 14 AGS Format Rule 9 8 8 5 21 AGS Format Rule 10a 6 6 6 18 AGS Format Rule 10c 6 6 4 16 FYI (Related to Rule 16) 8 8 8 24 AGS Format Rule 1: Has Non-ASCII character(s) (assuming that file encoding is 'utf-8'). (line 1146) AGS Format Rule 9: BGS_PROJ_ID not found in DICT group or the standard AGS4 dictionary. (line 2) AGS Format Rule 10a: Duplicate key field combination: DATA|HEADING|LOCA|LOCA_ORID (line 99) AGS Format Rule 10c: Parent entry for line not found in LOCA: X (line 1301) === Step 7: merge, repair and write a clean file === double-encoded degree signs to repair: 4 wrote output/clean/A417_missing_link_GI.ags (1,031 kB) with FILE/BGSFileSet01 beside it CRLF line endings: True downloads merged and rewritten repaired AGS Format Rule 2a 8866 0 0 FYI (Related to Rule 16) 24 8 8 AGS Format Rule 19a 21 8 0 AGS Format Rule 9 21 8 0 AGS Format Rule 19b 18 6 0 AGS Format Rule 10a 18 6 0 AGS Format Rule 10c 16 6 0 AGS Format Rule 7 14 5 0 AGS Format Rule 20 6 0 0 AGS Format Rule 1 4 4 0 left in the clean file: no rule errors FYI: HDPH_TYPE: Description of abbreviation "IP" is "Hand dug inspection pit" but it should be "Inspection pit" according to the standard abbreviations list. FYI: HDPH_TYPE: Description of abbreviation "WLS" is "Windowless sampler" but it should be "Dynamic (windowless) sampler" according to the standard abbreviations list. clean file reads back identically: True | load_clean() groups: 16 === Step 8: check the logs make geological sense === Units, youngest first: code TOPS Topsoil 66 MGR Made ground 85 HEAD Head 54 SLIP Landslide deposits 320 WHL White Limestone Formation 11 HMB Hampen Formation 21 GOG Great Oolite Group (undivided) 7 FE Fuller's Earth Formation 156 SALS Salperton Limestone Formation 63 ASLS Aston Limestone Formation 25 BLPL Birdlip Limestone Formation 322 IOG Inferior Oolite Group (undivided) 1 BDS Bridport Sand Formation 101 WHM Whitby Mudstone Formation 125 MRB Marlstone Rock Formation 12 DYS Dyrham Formation 109 CHAM Charmouth Mudstone Formation 20 NR No recovery or not classified 66 112 holes with GEOL, 90 reach bedrock. Holes needing a second look: CP217: bedrock out of order: WHM > MRB > DYS > WHM > MRB > DYS CPT202: gap or overlap of -0.40 m CP217, top to bottom: GEOL_TOP GEOL_BASE GEOL_GEOL UNIT GEOL_DESC 0.00 0.30 TOP TOPS Brown slightly sandy silty subangular and subrounded fi... 0.30 1.00 SLIP SLIP Soft orangish brown slightly sandy gravelly clayey SILT... 1.00 1.60 SLIP SLIP Soft brownish orange slightly sandy gravelly silty CLAY... 1.60 3.90 SLIP SLIP Loose brownish yellow clayey sandy subangular and subro... 3.90 4.55 SLIP SLIP Dense light brown slightly clayey sandy subangular and ... 4.55 5.45 SLIP SLIP Weak light brown bioclastic LIMESTONE locally subangula... 5.45 6.60 SLIP_LIAS SLIP Firm fissured orangish brown mottled grey clayey SILT. ... 6.60 8.85 SLIP_LIAS SLIP Firm fissured locally thinly laminated dark brown mottl... 8.85 9.50 SLIP_LIAS SLIP Firm fissured locally thinly laminated dark grey slight... 9.50 10.10 Lias Group WHM Very stiff fissured thinly laminated dark grey CLAY wit... 10.10 13.65 Lias Group WHM Very stiff fissured thinly laminated greenish grey mott... 13.65 14.60 Lias Group WHM Extremely weak thinly laminated greenish grey MUDSTONE ... 14.60 15.50 Lias Group MRB Medium strong grey LIMESTONE with thin and thick beds o... 15.50 16.90 Lias Group DYS Extremely weak thinly laminated greenish grey and dark ... 16.90 17.85 Lias Group DYS Extremely weak thinly laminated brownish grey mottled g... 17.85 19.10 Lias Group DYS Extremely weak thinly and thickly laminated dark grey a... 19.10 20.75 Lias Group WHM Extremely weak thinly laminated brownish grey mottled d... 20.75 21.85 Lias Group MRB Medium strong locally strong dark grey LIMESTONE with r... 21.85 25.30 Lias Group DYS Extremely weak thinly and thickly laminated greenish gr... 25.30 27.35 Lias Group DYS Extremely weak thinly and thickly laminated brownish gr... 27.35 27.80 Lias Group DYS Strong grey and dark grey LIMESTONE. Fractures are 10de... 27.80 28.85 Lias Group DYS Very stiff thinly and thickly laminated orangish brown ... 28.85 29.30 Lias Group DYS Extremely weak thinly laminated dark grey SILTSTONE wit... 29.30 30.80 Lias Group DYS Extremely weak thickly laminated dark grey SILTSTONE. L... 30.80 32.30 Lias Group DYS Very weak thinly laminated dark grey MUDSTONE. Rare thi... 32.30 33.50 Lias Group DYS Very weak thinly and thickly laminated dark grey MUDSTO... figures written: 01-group-rows.png, 02-validation-stages.png, 03-strip-logs.png, 04-cp217-repeat.png