Text-prompted Segment Anything on aerial imagery: Grounding DINO and SAM 2.1 in Python
Segment buildings, trees, water and cars from a text prompt with Grounding DINO and SAM 2.1 in Python on a CPU, the open equivalent of ArcGIS Pro's Text SAM. Choose the prompt wording, tile size and threshold on one Utrecht district, then measure accuracy on another against the Dutch BGT register.
- Level
- Advanced
- Tested with
- Python 3.12
- Source code
- Download scripts
Key takeaways
- Grounding DINO turns a text prompt into boxes, and SAM 2.1 turns each box into an outline. Both are Apache-2.0 and run on a CPU: about 4 seconds per prompt per tile, 4 seconds to encode a 1024-pixel tile and 32 ms per box.
- Settings chosen on Kanaleneiland, Utrecht, held up on Parkwijk, 4 km away: building pixels F1 0.70 (IoU 0.54), the same as the best Parkwijk could reach with its own settings. Tree crowns scored F1 0.66.
- The words matter: 'a tree.' beat 'tree.', and cutting the image into 512-pixel tiles found more small objects. SAM's outlines lifted building IoU from 0.38 for the boxes alone to 0.54.
- It finds whole roofs, not addresses: of 440 Parkwijk buildings under 150 m², mostly terraced houses and sheds, it matched 7. Water scored F1 0.40, because flat roofs and a paved square were called water.
Before you start
- Segment Anything on satellite and aerial imagery in Python: a samgeo tutorial, which introduces SAM, image embeddings, box prompts and IoU
- Comfortable with NumPy, rasterio and GeoPandas
- Python 3.10+ with the CPU build of PyTorch, transformers, rasterio, geopandas and matplotlib (see requirements.txt)
- About 3 GB of free disk space for PyTorch and the two models (1.7 GB together), and an internet connection; no GPU, account or token needed
Short answer: to segment aerial imagery from a text prompt in Python, load Grounding DINO (IDEA-Research/grounding-dino-base) and SAM 2.1 (facebook/sam2.1-hiera-large) with Hugging Face transformers. Cut the image into tiles, ask Grounding DINO for boxes matching a prompt such as "house.", and pass each box to SAM 2.1, which returns the object’s outline. Convert the masks to polygons with rasterio.features.shapes, using each tile’s geotransform. Then choose the prompt wording, tile size and score threshold on one area with good reference data, and check them on another before you trust the results.
Segment Anything (SAM) outlines objects very well, but it does not know what they are. In our samgeo tutorial, we told it where to look with points and boxes. In practice, you often have neither: you want “every building” or “every tree” in an image. A text prompt fills that gap. An open-vocabulary detector, Grounding DINO, reads the prompt and draws boxes; SAM turns each box into a precise outline. Esri packages the same two models in ArcGIS Pro as Text SAM.
The hard part is not running the models; it is knowing how far to trust them. This tutorial works on two 512 × 512 m districts of Utrecht in the Netherlands, where the national topographic register, the BGT, records every building, every stretch of water and every council tree. We choose the settings on one district and measure them on the other.
You will learn how to:
- download 25 cm aerial photography and matching reference data from the Dutch national services, PDOK and the BGT;
- run Grounding DINO on overlapping tiles so every object is counted once;
- turn every box into a SAM 2.1 mask, encoding each tile once, and georeference the masks as polygons;
- calibrate the prompt wording, tile size and score threshold on one district;
- test them on a second district with pixel, object and point-based scores, and find where they fail.
All outputs below are real results from text_prompt_sam.py, run on 20 November 2026 with Python 3.12, PyTorch 2.15.0 (CPU) and transformers 5.23.0. The machine was a desktop AMD Ryzen 9 7950X, limited to 8 threads, with no GPU. The first run takes a little over an hour, mostly in Grounding DINO, because it tries six prompts at two tile sizes on both sites. It caches the boxes and masks in data/, so later runs take a minute or two.
Install the models for CPU
Install the CPU build of PyTorch first, then everything else:
python -m venv .venv-samtext.venv-samtext\Scripts\activate # on macOS or Linux: source .venv-samtext/bin/activatepip install torch==2.15.0 torchvision==0.29.2 --index-url https://download.pytorch.org/whl/cpupip install -r requirements.txt # transformers, rasterio, geopandas, matplotlib, ...Both models come from the Hugging Face Hub the first time you load them: 0.9 GB for Grounding DINO base and 0.9 GB for SAM 2.1 Hiera Large. Neither needs an account or token. Recent versions of transformers include both architectures, so there is no need to install the original research repositories, which need a C++ compiler for Grounding DINO.
Meta’s SAM 3 (November 2025) understands short text prompts by itself, with no separate detector. Its weights are gated on Hugging Face: you must accept the SAM License and wait for approval, and the model has 848 M parameters, so in practice it needs an NVIDIA GPU. Grounding DINO and SAM 2.1 are both Apache-2.0, need no approval and run on a CPU. The workflow and the accuracy checks below apply to SAM 3 unchanged.
Step 1: two districts of 25 cm aerial photography
PDOK publishes a new national aerial survey every year. The 2025 summer survey, at 25 cm, is available by WMS under CC BY 4.0. We request 2048 × 2048 pixels for each site, which is 512 × 512 m, and save them as GeoTIFFs in the Dutch national grid, RD New (EPSG:28992):
SITES = { "kanaleneiland": dict(name="Kanaleneiland", lonlat=(5.0990, 52.0720), role="calibration"), "parkwijk": dict(name="Parkwijk, Leidsche Rijn", lonlat=(5.0590, 52.0985), role="test"),}RES, PX = 0.25, 2048 # 2048 x 2048 pixels of 25 cm = 512 x 512 mWMS = "https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0"LAYER = "2025_ortho25" # summer 2025, 25 cm; "Actueel_ortho25" always points at the latest year...params = dict(service="WMS", version="1.3.0", request="GetMap", layers=LAYER, styles="", crs=CRS, bbox=",".join(f"{v:.2f}" for v in s["bounds"]), width=PX, height=PX, format="image/png")r = requests.get(WMS, params=params, timeout=300)with MemoryFile(r.content) as mem, mem.open() as img: # PDOK returns a JPEG whatever you ask for rgb = img.read()[:3]Kanaleneiland is a 1950s and 1960s district of flats in long blocks, with tree-lined streets, a canal and a park lake. We use it to choose the settings. Parkwijk, in Leidsche Rijn, was built after 2000. It has terraced houses, apartment blocks, a railway, canals and car parks. It is used only to test the settings.
=== aerial photography (PDOK Luchtfoto 2025 ortho 25 cm) ===Kanaleneiland calibration centre RD 135240, 453790 | 2048 x 2048 px at 0.25 m = 512 x 512 mParkwijk, Leidsche Rijn test centre RD 132510, 456750 | 2048 x 2048 px at 0.25 m = 512 x 512 mStep 2: the reference, from the BGT
The BGT is the Dutch large-scale topographic register. Councils, water boards and Rijkswaterstaat maintain it to about 20 cm, and it is open under CC0. PDOK serves it through an OGC API Features endpoint, one collection per feature type. We fetch four: buildings (pand), water (waterdeel), point features including trees (vegetatieobject_punt) and road surfaces (wegdeel).
BGT = "https://api.pdok.nl/lv/bgt/ogc/v1/collections/{}/items"RD_URI = "http://www.opengis.net/def/crs/EPSG/0/28992"
def bgt(collection: str, bounds, path: Path) -> gpd.GeoDataFrame: """Current features of one BGT collection in a box, from PDOK's OGC API Features (paged, cached).""" if not path.exists(): feats, url = [], BGT.format(collection) params = {"bbox": ",".join(map(str, bounds)), "bbox-crs": RD_URI, "crs": RD_URI, "limit": 1000, "f": "json"} while url: r = requests.get(url, params=params, timeout=300) page = r.json() feats += page["features"] url, params = next((l["href"] for l in page["links"] if l["rel"] == "next"), None), None gpd.GeoDataFrame.from_features(feats, crs=CRS).to_file(path, driver="GPKG") g = gpd.read_file(path) return g[g["eind_registratie"].isna()] # the API also returns every earlier version of each featureThe last line matters. The API returns the history of each feature as well as its current state: for Kanaleneiland, the water request returned seven records, only one of them current. A feature whose eind_registratie (end of registration) is empty is the current version.
We keep existing buildings (status “bestaand”), water at or above ground level, and points whose plus_type is “boom” (tree). The BGT records each terraced house as its own building, or pand. From the air, a terrace is one roof, so we also merge touching buildings into roof blocks:
pand = pand[pand["status"].eq("bestaand")] # existing buildings, not planned oneswater = water[water["relatieve_hoogteligging"].ge(0)] # nothing in culverts or tunnelstrees = trees[trees["plus_type"].eq("boom") & trees.within(site_box)]pand = pand.assign(geometry=pand.intersection(site_box)).explode(index_parts=False)pand = pand[pand.area >= 5].reset_index(drop=True)blocks = gpd.GeoDataFrame(geometry=list(shapely.get_parts(pand.buffer(0.05).union_all())), crs=CRS)blocks["geometry"] = blocks.buffer(-0.05)=== reference: BGT (Basisregistratie Grootschalige Topografie) ===Kanaleneiland buildings (pand) 318 | roof blocks 144 | water 1.70 ha | registered trees 887 | road and parking surface 8.50 haParkwijk, Leidsche Rijn buildings (pand) 464 | roof blocks 175 | water 2.18 ha | registered trees 533 | road and parking surface 9.57 ha
The trees are those Utrecht manages: street, park and school trees. Trees in private gardens are not in the register. That makes the tree reference complete for recall in public space, but a crown in a garden counts as a false positive. Keep that in mind when you read the tree precision.
Step 3: load Grounding DINO and SAM 2.1
GD_ID, SAM_ID = "IDEA-Research/grounding-dino-base", "facebook/sam2.1-hiera-large"gd_proc = AutoProcessor.from_pretrained(GD_ID)gd = AutoModelForZeroShotObjectDetection.from_pretrained(GD_ID).eval()sam_proc = Sam2Processor.from_pretrained(SAM_ID)sam = Sam2Model.from_pretrained(SAM_ID).eval()=== models ===IDEA-Research/grounding-dino-base: 232 M parameters | facebook/sam2.1-hiera-large: 217 M parameters | loaded in 1.8 s (both Apache-2.0)The 1.8 seconds is from the cache. The first run spends a few minutes downloading.
Step 4: boxes from text with Grounding DINO
Grounding DINO’s prompt is a list of short phrases, each ending in a full stop: "house." or "car. bus.". It returns boxes with a score between 0 and 1 for how well each one matches. On aerial imagery, the scores are low. On Kanaleneiland the highest “house.” box scored 0.60, and nine boxes in ten scored below 0.30, which is below the 0.35–0.4 defaults suggested for everyday photographs. So we keep everything above 0.10 and choose the threshold later from the data.
Two further choices matter, and neither has an obvious answer, so we try both:
- Wording. We try three phrasings for buildings (
"building.","house.","roof.") and three for trees ("tree.","a tree.","tree crown."). - Tile size. Grounding DINO resizes every image so its short side is 800 pixels. A 1024-pixel tile is shrunk, and a car becomes about 14 pixels long. A 512-pixel tile is enlarged, and the same car becomes about 28 pixels long. Smaller tiles mean more calls but bigger objects.
The tiles overlap by half. Each tile keeps only the boxes whose centre falls in its central core, so an object on a tile edge is counted once, by the tile that sees it whole:
CANDIDATES = {"building": ["building.", "house.", "roof."], "tree": ["tree.", "a tree.", "tree crown."], "water": ["water."], "car": ["car."]}DET_TILES = (1024, 512)FLOOR = 0.10 # keep every box above this; thresholds are chosen afterwards from the scoresMAX_BOX_FRAC = 0.5 # a box covering half a tile is "the whole scene", never one object
def core(o: int, t: int) -> tuple[int, int]: lo = 0 if o == 0 else o + t // 4 hi = PX if o + t == PX else o + t - t // 4 return lo, hi
def detect(img: Image.Image, prompt: str): inputs = gd_proc(images=img, text=prompt, return_tensors="pt") with torch.inference_mode(): out = gd(**inputs) res = gd_proc.post_process_grounded_object_detection( out, inputs.input_ids, threshold=FLOOR, text_threshold=FLOOR, target_sizes=[img.size[::-1]])[0] boxes, scores = res["boxes"], res["scores"] area = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1]) keep = area < MAX_BOX_FRAC * img.size[0] * img.size[1] boxes, scores = boxes[keep], scores[keep] keep = nms(boxes, scores, 0.5) # one box per object return boxes[keep], scores[keep]We send one class per call rather than "building. tree. water. car." in one prompt. Grounding DINO can take several phrases at once, but they then compete for the same boxes. One class per call keeps each class’s scores separate, so each class can have its own threshold.
Step 5: an outline for every box with SAM 2.1
SAM 2.1’s image encoder is the slow part, so we encode each 1024-pixel tile once and then decode all its boxes in batches of 64. Each mask is turned into polygons with the tile’s own geotransform, so it lands in the right place in RD New:
def segment(embeddings, boxes: list, batch: int = 64): masks, quality = [], [] for i in range(0, len(boxes), batch): inputs = sam_proc(input_boxes=[boxes[i:i + batch]], original_sizes=[[TILE, TILE]], return_tensors="pt") with torch.inference_mode(): out = sam(image_embeddings=embeddings, input_boxes=inputs["input_boxes"], multimask_output=False) masks.append(sam_proc.post_process_masks(out.pred_masks, [[TILE, TILE]])[0][:, 0].numpy().astype(bool)) quality.append(out.iou_scores[0, :, 0].numpy()) return np.concatenate(masks), np.concatenate(quality)
def to_polygon(mask: np.ndarray, transform, min_part_m2: float = 0.5): parts = [shape(g) for g, v in rasterio.features.shapes(mask.astype("uint8"), mask=mask, transform=transform) if v == 1] parts = [p for p in parts if p.area >= min_part_m2] return shapely.MultiPolygon(parts).simplify(RES / 2) if parts else None...t_tf = from_origin(s["bounds"][0] + ox * RES, s["bounds"][3] - oy * RES, RES, RES) # this tile's geotransformwith torch.inference_mode(): emb = sam.get_image_embeddings(sam_proc(images=tile, return_tensors="pt")["pixel_values"])multimask_output=False asks SAM for one mask per box. With a point, SAM’s three candidate masks help resolve ambiguity, such as a window or the whole house. A box already says how big the object is.
count total_s seconds_eachstepGrounding DINO, 1024-px tile 144 583.458 4.052Grounding DINO, 512-px tile 784 2981.202 3.803SAM 2.1 image encoder, 1024-px tile 18 77.068 4.282SAM 2.1 mask decoder, per box 14302 456.565 0.032Grounding DINO took 4 seconds per prompt per tile, whatever the tile size. SAM 2.1 took 4.3 seconds to encode a tile and 32 milliseconds per box. With the four settings chosen below, one 512 × 512 m site takes about 8 minutes on 8 CPU threads, mostly in Grounding DINO on 512-pixel tiles.
Figure 2 shows both the strength and the weakness. Where Grounding DINO’s box is right, SAM 2.1’s outline follows the roof edge almost exactly. The low-rise houses in the centre had boxes, shown dotted, but their scores were below the threshold, so SAM never saw them.
Step 6: choose the settings on Kanaleneiland
Each class needs its own way of scoring:
- Buildings: pixel F1 against the BGT buildings, meaning how much of the building area is right. We also match polygons one to one at IoU ≥ 0.5, against single
pandand against roof blocks. - Trees: the BGT has points, not crowns. A crown counts as correct if it holds a registered tree, and a registered tree counts as found if it falls inside a crown.
- Water: pixel F1 against the BGT water.
- Cars: there is no register of parked cars, so we check where the detections are. The share that lie on a BGT road or parking surface measures precision.
def score(cls: str, pred: gpd.GeoDataFrame, s) -> dict: ref, pred = s["ref"], clip(pred, s) if cls == "building": # headline: building area right, pixel by pixel; then object by object _, used_p, _ = match_objects(pred, ref["building"]) _, used_pb, _ = match_objects(pred, ref["block"]) return dict(pixel_scores(pred.geometry, ref["building"].geometry, s), objects=len(pred), **{f"obj_{k}": v for k, v in prf(int(used_p.sum()), len(pred), len(ref["building"])).items()}, block_f1=prf(int(used_pb.sum()), len(pred), len(ref["block"]))["f1"]) if cls == "tree": # a crown is right if it holds a registered tree; a tree is found if covered hit = gpd.sjoin(ref["tree"], pred[["geometry"]], predicate="within") p, r = hit["index_right"].nunique() / len(pred), hit.index.nunique() / len(ref["tree"]) return dict(precision=p, recall=r, f1=2 * p * r / (p + r) if p + r else 0.0, objects=len(pred)) if cls == "water": return dict(pixel_scores(pred.geometry, ref["water"].geometry, s), objects=len(pred)) road = ref["road"].geometry.iloc[0].buffer(1.0) return dict(objects=len(pred), on_road=float(pred.representative_point().within(road).mean()))We score every wording at both tile sizes and at thresholds from 0.10 to 0.50 in steps of 0.025, and keep the best F1 for each class on Kanaleneiland only:
best F1 on Kanaleneiland for each wording and tile size: cls prompt det_tile thr objects precision recall f1building building. 512 0.20 88 0.64 0.86 0.74building building. 1024 0.18 77 0.60 0.92 0.73building house. 512 0.22 99 0.67 0.85 0.75building house. 1024 0.22 76 0.64 0.84 0.73building roof. 512 0.15 156 0.55 0.87 0.67building roof. 1024 0.15 93 0.54 0.87 0.66 tree a tree. 512 0.10 427 0.73 0.71 0.72 tree a tree. 1024 0.10 262 0.81 0.68 0.74 tree tree crown. 512 0.10 461 0.66 0.71 0.68 tree tree crown. 1024 0.10 143 0.80 0.55 0.65 tree tree. 512 0.12 1076 0.51 0.80 0.62 tree tree. 1024 0.18 232 0.71 0.74 0.72 water water. 512 0.30 7 0.75 0.56 0.64 water water. 1024 0.15 9 0.71 0.61 0.66
Some findings are worth knowing before you write your own prompts:
"roof."sounds right for imagery seen from above, but it was the worst building prompt, with the lowest precision of the three (55%)."tree."on 512-pixel tiles returned 1,076 crowns, only half of which held a registered tree;"a tree."returned 427 and scored higher. On 1024-pixel tiles,"a tree."had the best F1 of all, 0.74.- The best thresholds were between 0.10 and 0.225. Grounding DINO’s scores for aerial imagery are low and differ by wording, so set the threshold by measuring rather than by habit.
For cars, the more we accept, the more land off the road:
objects on_roaddet_tile 512 1024 512 1024thr0.10 506.0 197.0 0.79 0.820.15 243.0 72.0 0.88 0.890.20 126.0 26.0 0.90 0.850.25 66.0 10.0 0.89 1.000.30 28.0 3.0 0.96 1.00We choose the setting that finds the most cars while keeping at least 95% on a road or parking surface. That is "car." on 512-pixel tiles at 0.30, which kept just 28 cars. That is a precise but very incomplete count: at 0.10, the same tiles gave 506 boxes, 79% of them on a road or parking surface. The 512-pixel tiles found 2.6 times as many cars as the 1024-pixel tiles at 0.10, which confirms that small objects need small tiles.
chosen: building: 'house.' on 512-px tiles, score >= 0.225; tree: 'a tree.' on 1024-px tiles, score >= 0.1; water: 'water.' on 1024-px tiles, score >= 0.15; car: 'car.' on 512-px tiles, score >= 0.3Step 7: test on Parkwijk
Now we apply exactly the same settings to Parkwijk and compare with the best each site could have reached with its own settings:
Kanaleneiland (calibration) building 99 polygons | pixels: precision 67%, recall 85%, IoU 0.60, F1 0.75 | objects (IoU >= 0.5): F1 0.12 against single BGT pand, 0.28 against roof blocks tree 262 crowns | 81% hold a registered tree; 68% of 887 registered trees are inside a crown; F1 0.74 water 9 polygons | pixels: precision 71%, recall 61%, IoU 0.49, F1 0.66 car 28 cars | 96% on a BGT road or parking surface best F1 this site could have reached with its own settings: building 0.75, tree 0.74, water 0.66
Parkwijk, Leidsche Rijn (test) building 103 polygons | pixels: precision 59%, recall 86%, IoU 0.54, F1 0.70 | objects (IoU >= 0.5): F1 0.08 against single BGT pand, 0.35 against roof blocks tree 242 crowns | 65% hold a registered tree; 67% of 533 registered trees are inside a crown; F1 0.66 water 11 polygons | pixels: precision 42%, recall 38%, IoU 0.25, F1 0.40 car 33 cars | 94% on a BGT road or parking surface best F1 this site could have reached with its own settings: building 0.70, tree 0.68, water 0.49
Buildings transferred well. On Parkwijk, 86% of the building area was found, with a pixel F1 of 0.70 and IoU of 0.54. That equals the best Parkwijk could have reached with any wording, tile size or threshold, so tuning on Kanaleneiland lost nothing. Trees dropped from 0.74 to 0.66. Parkwijk was built after 2000, so its street trees are younger, with smaller crowns. Water fell from 0.66 to 0.40.
Figure 5 shows why the building threshold transferred: both curves peak at the same place. Tree F1 fell to 0.50 on Kanaleneiland and 0.25 on Parkwijk by a threshold of 0.2. At 0.35, a common default for everyday photographs, no trees were found at all.
What SAM adds
Are SAM’s outlines worth the extra step, or would Grounding DINO’s boxes do? We score the boxes themselves as building outlines:
Kanaleneiland boxes: pixel IoU 0.33, F1 0.50, object F1 0.04 | SAM 2.1 masks: pixel IoU 0.60, F1 0.75, object F1 0.12Parkwijk, Leidsche Rijn boxes: pixel IoU 0.38, F1 0.55, object F1 0.04 | SAM 2.1 masks: pixel IoU 0.54, F1 0.70, object F1 0.08SAM lifted the building IoU from 0.33 to 0.60 on Kanaleneiland and from 0.38 to 0.54 on Parkwijk. Most of Utrecht’s buildings are not aligned with the image grid, so a box includes a lot of street and garden around each one.
Where it fails
The object scores tell a different story from the pixel scores: F1 0.08 against single buildings and 0.35 against roof blocks. On Parkwijk, the chosen prompt found most of the building area but few individual buildings:
buildings recallsizeunder 20 m² 200 0.0020-50 m² 46 0.0450-150 m² 194 0.03150-500 m² 12 0.58over 500 m² 12 0.67median IoU of the matched buildings: 0.67
Figure 6 shows two failure modes:
- Terraces. Grounding DINO draws one box around a row of eight houses, and SAM outlines the whole row. The area is right, but none of the eight single houses matches. This is a property of the image, not a bug: from above, a terrace is one roof.
- Sheds and garages. Of 464 BGT buildings in Parkwijk, 200 are smaller than 20 m². None was found. At 25 cm a 10 m² shed is about 12 × 12 pixels, below what Grounding DINO detects reliably on a 512-pixel tile.
Across the 440 Parkwijk buildings under 150 m², it matched 7. If you need address-level footprints, start from boxes drawn around existing footprints, as in our samgeo tutorial, or split SAM’s roof outlines by cadastral parcels.
Water failed in a different way. On Parkwijk, 1.12 ha of the 1.95 ha marked as water was not water: three flat roofs (0.42 ha) and the paved square in front of the station, which you can see shaded blue in Figure 3. On Kanaleneiland, the false water was dark shadow beside blocks of flats, a shaded road and a pale rectangle of hard surface by the park. Much of the narrow, shaded canal network was missed. Grounding DINO knows what “water” looks like in a photograph of a lake, not in a vertical view of a canal between trees. For water, a spectral index on near-infrared imagery or the BGT itself is more reliable.
Cars kept their precision on Parkwijk, 94% on a road or parking surface, but 33 cars is a small share of those parked there. For counting cars, use a detector trained on aerial images of vehicles.
Step 8: save the polygons
The chosen polygons go into one GeoPackage, one layer per site and class, with Grounding DINO’s box score, SAM’s own quality estimate (sam_iou) and the box itself:
for key in SITES: for cls, g in final[key].items(): if len(g): g.drop(columns="box").to_file(gpkg, layer=f"{key}_{cls}", driver="GPKG")wrote 787 polygons in 8 layers to text_prompt_sam.gpkg (EPSG:28992)Open text_prompt_sam.gpkg in QGIS over the PDOK Luchtfoto WMS to inspect the results. Style by score to see which polygons Grounding DINO was unsure about. metrics.json holds every score above.
Limitations and how to improve them
- Scores are not probabilities. A score of 0.22 for “house.” means nothing outside this wording, tile size and imagery. Calibrate on an area with good reference data, test on another, and keep the threshold with the prompt.
- Two sites, one city. Both sites are Dutch suburbs photographed on the same summer survey. Expect different results for city centres, rural areas, winter imagery or other countries’ building styles. Calibrate again on a sample of your own area.
- CPU speed. About 8 minutes per 0.26 km² for four classes means about half an hour per km². A GPU makes this practical for whole towns; on a CPU, ask for one class at a time and only on the tile size it needs.
- Larger tiles for big objects. Large flats and the lake were found on both tile sizes; small houses, cars and young trees needed 512 pixels. A town-wide run could use 1024-pixel tiles for large classes and 512 for small ones.
- No cross-tile merging. Each object is assigned to the tile whose core holds its box centre, but a very large building that crosses several tiles can still be split. Merging overlapping polygons of the same class after the run fixes most cases.
- Reference data has its own rules. The BGT draws buildings at the walls, while the camera sees the roof with its overhang and lean. That limits IoU even for a perfect outline of the roof; the matched buildings had a median IoU of 0.67.
Frequently asked questions
Can Segment Anything segment from a text prompt?
The original SAM and SAM 2 cannot: they accept points, boxes and masks. The usual solution is to add an open-vocabulary detector such as Grounding DINO, which turns the text into boxes for SAM. This is what samgeo’s LangSAM and ArcGIS Pro’s Text SAM do. SAM 3, released by Meta in November 2025, accepts short text prompts directly, but its weights are gated and it needs a GPU in practice.
What is the open-source equivalent of ArcGIS Pro’s Text SAM?
Text SAM is Grounding DINO followed by SAM, packaged as an ArcGIS deep learning package (DLPK). In Python, you can build the same chain with Hugging Face transformers as in this tutorial, or use samgeo’s LangSAM class. In QGIS, the GeoAI, SamGeo and GeoOSAM plugins offer text prompts.
What prompt should I use for buildings in aerial imagery?
On our Utrecht test, "house." scored highest (pixel F1 0.75), with "building." close behind (0.74). "roof." was clearly worse (0.67). The best wording depends on the area and imagery, so test two or three wordings against a reference before running a large area.
How accurate is Grounding DINO and SAM on aerial imagery?
On a district of Utrecht not used for tuning, with 25 cm imagery, it found 86% of the building area with a pixel F1 of 0.70 and IoU of 0.54. It found 67% of the registered street and park trees inside a crown, with crown F1 0.66, and water scored F1 0.40. It matched only 7 of 440 individual buildings under 150 m², because it outlines whole terraces and misses small sheds.
Can I run Grounding DINO and SAM 2 without a GPU?
Yes. On 8 CPU threads, Grounding DINO base took about 4 seconds per prompt per tile, SAM 2.1 Hiera Large took 4.3 seconds to encode a 1024-pixel tile and 32 ms per box. Four classes on a 512 × 512 m site took about 8 minutes. That is fine for testing and small areas; for whole towns, use a GPU.
Why are Grounding DINO’s scores so low on satellite and aerial imagery?
Grounding DINO was trained on everyday photographs, where a “house” is seen from the side. From above, it is less sure, and correct boxes scored 0.2–0.35 rather than 0.5 or more. That is why we kept every box above 0.10 and chose each threshold from a reference.
Next steps
- Start with points and boxes, automatic segmentation and samgeo in Segment Anything on satellite and aerial imagery in Python.
- Read our overview of Segment Anything for GIS: SAM, SAM 2 and SAM 3, the tools in Python, ArcGIS Pro and QGIS, and how to check the results.
- Get building footprints and heights from lidar instead of imagery in Extract building footprints and heights from LiDAR in Python.
- Train a model on your own labels rather than prompting a foundation model in Train a U-Net for satellite image segmentation in PyTorch.
- Turn other rasters into clean polygons in Raster to vector in Python.
Data and licences
- Aerial photography: Luchtfoto 2025 Ortho 25 cm RGB, Beeldmateriaal Nederland, served by PDOK under CC BY 4.0. Accessed 20 November 2026.
- Reference data: Basisregistratie Grootschalige Topografie (BGT), the BGT source holders and Kadaster, via the PDOK BGT OGC API, under CC0 1.0. Accessed 20 November 2026.
- Grounding DINO (
IDEA-Research/grounding-dino-base): IDEA Research, Apache-2.0 licence. Liu et al. (2023), Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, arXiv:2303.05499. - SAM 2.1 (
facebook/sam2.1-hiera-large): Meta AI, Apache-2.0 licence. Ravi et al. (2024), SAM 2: Segment Anything in Images and Videos, arXiv:2408.00714. - Code:
text_prompt_sam.py, tested with Python 3.12.0, PyTorch 2.15.0 (CPU), transformers 5.23.0, rasterio 1.5.2, geopandas 1.2.0 and shapely 2.2.0.
Key terms
- Grounding DINO
- An open-vocabulary object detector from IDEA Research (2023, Apache-2.0). Given an image and a text prompt, it returns boxes for the objects the text describes, each with a score.
- SAM 2 and SAM 2.1
- Meta's second Segment Anything Model (2024, Apache-2.0), with a faster Hiera image encoder and a memory that tracks objects through video. SAM 2.1 is the improved checkpoint released in September 2024.
- Text SAM
- Esri's name for a deep learning package that chains Grounding DINO and SAM in ArcGIS Pro. This tutorial builds the same chain in Python, with SAM 2.1 in place of the original SAM.
- BGT
- Basisregistratie Grootschalige Topografie: the Dutch national large-scale topographic register, with every building, road, water body and, where councils record them, every tree, at about 20 cm accuracy. Open data under CC0.
- Pixel F1 and IoU
- Pixel scores compare the area marked as an object with the area in the reference. F1 is the harmonic mean of precision and recall; IoU is the shared area over the combined area. F1 = 2 IoU / (1 + IoU).
- Calibration and test sites
- Settings are chosen on one site and only measured on another. A score on the site used for tuning is optimistic; the test site shows what to expect elsewhere.
Data for this tutorial
- PDOK (Publieke Dienstverlening Op de Kaart) · Luchtfoto 2025 25 cm aerial photography by WMS
- BGT (Kadaster) · buildings, water, trees and roads used as the reference, by OGC API Features
Licences and formats for each source are listed in the GIS data sources catalogue.
Need this done on your own data? TechnicalGIS takes on consulting projects like this one, from a single analysis to a pipeline your team can rerun.