---
title: "Segment Anything for GIS: SAM, SAM 2 and SAM 3 on aerial and satellite imagery"
description: "How Segment Anything works on aerial imagery: prompts, Python, ArcGIS Pro and QGIS tools, masks to polygons, accuracy checks and where it fails."
url: https://www.technicalgis.com/articles/segment-anything-for-gis/
author: Sean Blaney (https://www.technicalgis.com/about/#sean-blaney)
publisher: TechnicalGIS (https://www.technicalgis.com/about/)
published: 2026-11-20
updated: 2026-11-20
language: en-GB
---

# Segment Anything for GIS: SAM, SAM 2 and SAM 3 on aerial and satellite imagery

> How Segment Anything works on aerial imagery: prompts, Python, ArcGIS Pro and QGIS tools, masks to polygons, accuracy checks and where it fails.

## Key takeaways

1.  Segment Anything draws outlines; it does not name what it outlines. Points, boxes or a text prompt tell it where to look. SAM and SAM 2 need a separate detector, such as Grounding DINO, for text; SAM 3 (November 2025) reads short text prompts by itself.
2.  SAM, SAM 2 and SAM 2.1 are Apache-2.0 and run on a CPU. SAM 3 and 3.1 are released under Meta's own SAM License, with gated weights, and in practice need an NVIDIA GPU.
3.  In our tests, boxes from an existing map gave the best outlines: a median IoU of 0.71 against OpenStreetMap houses, 145 of 146 above 0.5. A text prompt, tuned on one Utrecht district, found building area on another at pixel F1 0.70.
4.  Measure the results against a reference and choose the score that matches the question. On the same polygons, building area scored F1 0.70 but individual buildings only 0.08, because terraces came out as one roof and sheds were missed.

Meta’s Segment Anything models changed what it takes to outline things in imagery. Before 2023, extracting buildings, trees or ponds from aerial photographs meant training a model on thousands of labelled examples, or digitising by hand. SAM outlines almost any object in a fraction of a second once you tell it where to look, with no training at all.

That makes it very useful for GIS work, but it is easy to overestimate. SAM finds edges well, but it does not know what a building is. It works on 1024-pixel images, not on maps. And its results need checking like any other classification. This article explains what the models are, how to prompt them, which tools bring them into Python, ArcGIS Pro and QGIS, and how to measure the results. The figures come from two tutorials we ran end to end:

-   [Segment Anything on satellite and aerial imagery in Python](https://www.technicalgis.com/tutorials/segment-anything-satellite-imagery/) uses samgeo on 0.5 m NAIP imagery of Ames, Iowa, with automatic, point and box prompts, measured against OpenStreetMap.
-   [Text-prompted Segment Anything on aerial imagery](https://www.technicalgis.com/tutorials/text-prompt-segment-anything-grounding-dino-sam2/) uses Grounding DINO and SAM 2.1 on 25 cm Dutch aerial photography, with text prompts for buildings, trees, water and cars, measured against the Dutch national topographic register, the BGT.

## The models: SAM, SAM 2 and SAM 3

| Model | Released | Works on | Prompts | Licence | Runs on a CPU? |
| --- | --- | --- | --- | --- | --- |
| SAM | April 2023 | images | points, boxes, masks | Apache-2.0 | yes, with ViT-B (94 M parameters) |
| SAM 2 and 2.1 | July and September 2024 | images and video | points, boxes, masks | Apache-2.0 | yes; Hiera Large is 217 M parameters |
| SAM 3 | November 2025 | images and video | text, example boxes, points, boxes, masks | SAM License (gated) | not in practice; 848 M parameters |
| SAM 3.1 | March 2026 | images and video | as SAM 3 | SAM License (gated) | as SAM 3 |

**SAM** (Kirillov et al., 2023) has a heavy image encoder, which turns a 1024 × 1024 image into an embedding once, and a light decoder, which turns each prompt into a mask in about 30 milliseconds. It was trained on SA-1B: 11 million photographs with 1.1 billion masks. On a CPU, the smallest version, ViT-B, encoded a tile in about 4 seconds in our first tutorial.

**SAM 2** (Ravi et al., 2024) replaced the encoder with a faster one, Hiera, and added a memory that follows an object from one video frame to the next. The September 2024 checkpoints, SAM 2.1, are the ones to use. For GIS, the video memory matters when you treat a time series of images as frames: an object prompted once in the first year can be followed through the rest.

**SAM 3** (November 2025) adds what the earlier models lacked: it accepts a short noun phrase, such as “solar panel”, or an example box, and returns every matching object in the image. Meta calls this promptable concept segmentation. **SAM 3.1** (March 2026) is a drop-in update that tracks many objects in video faster. Both are released under Meta’s SAM License rather than Apache-2.0. The weights are gated: you request access on Hugging Face and accept the licence before you can download them.

## How to prompt it

SAM always needs to be told where to look. The options, from the most control to the least:

-   **Boxes.** A box around one object is the most reliable prompt, because it also says how big the object is. Boxes can come from an existing map, a detector or a person dragging a rectangle. In our first tutorial, boxes drawn around OpenStreetMap footprints gave a median IoU of 0.71, and 145 of 146 houses scored above 0.5.
-   **Points.** A click inside the object, optionally with clicks outside it to exclude areas. One point is ambiguous (a chimney, a roof plane or the whole house?), so SAM returns three candidate masks with its own quality estimate for each. Points suit interactive digitising.
-   **Automatic.** SAM places a grid of points over the image and keeps the best, non-overlapping masks. It outlines everything, roofs, trees, car parks and playing fields, without saying which is which. You then need rules or a classifier to keep the objects you want. In our first tutorial, automatic masks outlined 73% of the OpenStreetMap buildings, but after rules on area, shape and NDVI to separate buildings from everything else, 55% remained.
-   **Text.** With SAM and SAM 2, a text prompt goes to an open-vocabulary detector, usually Grounding DINO, which returns boxes, and the boxes go to SAM. With SAM 3, the text goes to the model directly.

![Left: a zoomed area of houses around two cul-de-sacs, with dashed yellow OpenStreetMap outlines and solid red SAM outlines that follow each roof closely. Right: a histogram of IoU between SAM masks and OpenStreetMap footprints for 146 buildings, clustered between 0.6 and 0.85 with a median of 0.71.](https://www.technicalgis.com/_astro/06-box-prompts-iou.DgDkhFWB_Z1lKljC.webp)

_Box prompts from OpenStreetMap footprints on NAIP imagery of Ames, Iowa: SAM's outlines (red) against the footprints (dashed yellow). From our samgeo tutorial._

### Text prompts need calibrating

Text prompts are the most convenient, and the least predictable. In the text-prompt tutorial, Grounding DINO’s scores on aerial imagery were low: nine boxes in ten scored below 0.30, under the 0.35–0.4 defaults suggested for everyday photographs. At 0.35, it found no trees at all. The wording mattered too. For buildings, “house.” scored pixel F1 0.75, “building.” 0.74 and “roof.” 0.67. For trees, “a tree.” beat “tree.”. Cutting the image into smaller tiles, 512 pixels rather than 1024, found 2.6 times as many cars, because Grounding DINO enlarges a small tile and the cars become bigger.

The safe way to use a text prompt is to choose the wording, tile size and threshold on an area where you have a reference, and then check them on a second area before running everything else. The settings we chose on Kanaleneiland in Utrecht gave a building F1 of 0.70 on Parkwijk, 4 km away, the same as the best Parkwijk could reach with its own settings.

![Two districts of Utrecht with every polygon from four text prompts: buildings in orange, tree crowns in green, water in blue and a few cars in yellow. In Parkwijk, a paved square beside the railway is also shaded blue.](https://www.technicalgis.com/_astro/03-four-prompts-both-sites.O4_6sMSS_Z1vzNIL.webp)

_Four text prompts, with settings chosen on Kanaleneiland (left) and tested on Parkwijk (right). The blue square beside the railway in Parkwijk is paving, not water. From our text-prompt tutorial._

## The tools

### Python

-   **[segment-geospatial (samgeo)](https://samgeo.gishub.org/)**, by Qiusheng Wu, MIT licence. It wraps SAM, SAM 2 and SAM 3 for geospatial data: it reads GeoTIFFs and web map tiles, accepts prompts in map coordinates, writes masks as GeoTIFF or vector, and includes `LangSAM` for text prompts with Grounding DINO. SAM 3 support is installed with the `segment-geospatial[samgeo3]` extra. This is the quickest start.
-   **[Hugging Face transformers](https://huggingface.co/docs/transformers/)** includes Grounding DINO, SAM, SAM 2 and SAM 3 as standard classes. You handle tiling and georeferencing yourself, but you control every step and can run on a CPU without compiling anything. Our text-prompt tutorial uses this route.
-   **Meta’s own repositories** ([sam2](https://github.com/facebookresearch/sam2), [sam3](https://github.com/facebookresearch/sam3)) have the reference code, the video predictors and the newest features first.

### ArcGIS Pro

Esri publishes two ready-made deep learning packages on ArcGIS Living Atlas, used with the **Detect Objects Using Deep Learning** tool in the Image Analyst extension:

-   **[Segment Anything Model (SAM)](https://www.arcgis.com/home/item.html?id=9b67b441f29f4ce6810979f5f0667ebe)** (May 2023) outlines everything in the image, as SAM’s automatic mode does.
-   **[Text SAM](https://www.arcgis.com/home/item.html?id=8df3bf4167bc4c7b967f677f8b362ec3)** (March 2024) chains Grounding DINO and SAM: you type the object classes, and it returns a feature class of polygons. Its [source code is on GitHub](https://github.com/Esri/raster-deep-learning/tree/master/samples/TextSAM).

Esri recommends a GPU with at least 8 GB of memory for both, and they take 8-bit, 3-band imagery. Both packages need the ArcGIS deep learning libraries installed into Pro’s Python environment.

### QGIS

-   **GeoAI plugin** (opengeos, MIT licence), by the samgeo author, brings samgeo’s SAM, SAM 2 and SAM 3 tools and other GeoAI models into QGIS. It is the successor its author now recommends over the earlier SamGeo plugin.
-   **Geo-SAM** (MIT licence) encodes an image once, ahead of time, and then lets you digitise by clicking points or dragging boxes, with each mask appearing almost at once. It handles points and boxes only.
-   **GeoOSAM** (GPL v2) supports SAM 2.1 and SAM 3, including free SAM 3 text prompts on the visible map extent; automatic tiling over larger areas is a paid feature.

Interactive plugins suit digitising, where a person checks every outline as it appears. For unattended extraction over a large area, a script is easier to reproduce and to measure.

## From masks to GIS features

A mask is a grid of pixels, so turning it into a GIS feature follows the same steps as any raster-to-vector conversion:

1.  **Keep the geotransform of each tile.** SAM sees a plain 1024 × 1024 image. If you cut a GeoTIFF into tiles, each tile’s origin must be carried through, or every polygon will land at the corner of the map.
2.  **Polygonise** with `rasterio.features.shapes` (or Raster to Polygon in ArcGIS, Polygonize in QGIS), drop specks below a minimum area, and simplify by about half a pixel to remove the staircase edges.
3.  **Handle tile edges.** Objects cut by a tile edge come out in pieces. Overlap the tiles and either keep each object only in the tile whose centre region holds it, or merge overlapping polygons of the same class afterwards.
4.  **Keep the scores.** Store the detector’s score and SAM’s own quality estimate with each polygon, so you can filter or review the doubtful ones later.

Our [raster to vector tutorial](https://www.technicalgis.com/tutorials/raster-to-vector-python/) covers the polygonising and cleaning in more depth.

## How to check the results

A SAM result looks convincing on screen, which makes measuring it more important, not less. Three things decide what a score means.

**The reference.** OpenStreetMap is free everywhere but incomplete; national registers such as the Dutch BGT or Ordnance Survey MasterMap are complete but follow their own rules. The BGT draws buildings at the walls, while the camera sees the roof with its overhang; it records each terraced house separately, but from above a terrace is one roof. Utrecht registers every street and park tree, but not trees in private gardens, so a correct garden tree counts as a false positive.

**The score.** Choose the measure that answers your question:

| On the Parkwijk test site | Score |
| --- | --- |
| Building area, pixel by pixel | F1 0.70 (IoU 0.54) |
| Individual BGT buildings, matched one to one at IoU ≥ 0.5 | F1 0.08 |
| Roof blocks (touching buildings merged), matched one to one | F1 0.35 |
| Registered trees inside a predicted crown | 67% of 533 |
| Water area, pixel by pixel | F1 0.40 |

All five rows describe the same run. If you want to know how much of a district is roofed, it did well. If you want one polygon per address, it did badly.

**Calibration and test.** Choosing the prompt and threshold on the same area you score inflates the result. Score on a second area that played no part in the choice.

## Where it fails

Both tutorials found the same kinds of failure:

-   **Small objects.** In Parkwijk, none of the 200 buildings under 20 m² was found, and across all 440 buildings under 150 m² the text prompt matched 7. At 25 cm, a 10 m² shed is about 12 × 12 pixels. Small tiles help, but only so far.
-   **Joined objects.** A terrace of eight houses, or a school with several roof levels, comes out as one mask or as several pieces that match nothing. In Ames, SAM split a school’s flat roof at its parapets. In Utrecht, Grounding DINO drew one box around a whole terrace.
-   **Look-alikes.** Text prompts follow appearance, not meaning. “water.” matched three flat roofs and a paved station square in Parkwijk, and dark shadow beside blocks of flats in Kanaleneiland. Pixel F1 for water was 0.40 on the test site.
-   **Trees over buildings.** Houses partly under trees gave masks that included the canopy, or were dropped by a vegetation rule.
-   **Anything not like a photograph.** SAM was trained on ordinary colour photographs, and both of our tests used RGB aerial imagery. Radar, elevation and single-band data look very different; render them as an image with clear edges, such as a hillshade, before prompting, and measure the result against a reference.

![Parkwijk with polygons coloured by match: large apartment blocks in green, whole terraces of houses in orange with thin red outlines of the individual houses inside, and small red squares of garden sheds that have no polygon.](https://www.technicalgis.com/_astro/06-parkwijk-building-errors.B6ddSCGK_Z1Cf29V.webp)

_Parkwijk building errors from the text prompt 'house.': green polygons match one BGT building, orange ones match none, and red outlines are BGT buildings not matched. The terraces are found as whole rows._

## When to use it, and when not

Segment Anything is a good choice when:

-   you have boxes or points already, from a map, a detector or a person, and want accurate outlines;
-   you are digitising interactively and want each click to give a clean polygon;
-   you have no labelled training data and need a first result to check and correct.

Other tools are often better when:

-   you need one feature per address or per tree: start from existing footprints or train a model on your own labels, as in our [U-Net tutorial](https://www.technicalgis.com/tutorials/train-unet-satellite-segmentation-pytorch/);
-   you have lidar: [building footprints and heights from lidar](https://www.technicalgis.com/tutorials/building-footprints-lidar-python/) are more complete than anything traced from a photograph;
-   you need land cover for every pixel rather than objects: a [classifier on Sentinel-2](https://www.technicalgis.com/tutorials/land-cover-classification-random-forest/) answers that directly;
-   you need to map water: a spectral index on near-infrared bands separates water from paving and shadow, which a text prompt on RGB did not.

## Frequently asked questions

### What is Segment Anything in GIS?

It is Meta’s family of image segmentation models, SAM, SAM 2 and SAM 3, used on aerial, drone and satellite imagery to outline features such as buildings, trees, fields and water. You prompt it with points, boxes or text, and convert the resulting masks to polygons with the image’s geotransform.

### Which Segment Anything model should I use?

For box and point prompts on a CPU, SAM 2.1 is the best choice: it is Apache-2.0, faster than the original SAM and more accurate. For text prompts without a GPU, pair SAM 2.1 with Grounding DINO. If you have an NVIDIA GPU and accept the SAM License, SAM 3 reads text prompts directly and returns every matching object.

### Does ArcGIS Pro support Segment Anything?

Yes. Esri publishes a Segment Anything Model package and a Text SAM package, which pairs Grounding DINO with SAM, on ArcGIS Living Atlas. Both run with the Detect Objects Using Deep Learning tool in the Image Analyst extension, and Esri recommends a GPU with at least 8 GB of memory.

### Is there a QGIS plugin for Segment Anything?

Several. The GeoAI plugin brings samgeo’s SAM, SAM 2 and SAM 3 tools into QGIS; Geo-SAM offers fast interactive digitising with points and boxes; and GeoOSAM supports SAM 2.1 and SAM 3, including text prompts.

### How accurate is Segment Anything on aerial imagery?

It depends on the prompt and on what you measure. With boxes from OpenStreetMap footprints, SAM’s outlines of houses had a median IoU of 0.71. With the text prompt “house.”, on a district not used for tuning, building area scored pixel F1 0.70, but only 22 of its 464 individual buildings were matched one to one, because terraces came out as one roof and sheds were missed.

### Can I use Segment Anything commercially?

SAM, SAM 2 and SAM 2.1 are released under Apache-2.0, which allows commercial use. Grounding DINO is also Apache-2.0. SAM 3 and SAM 3.1 use Meta’s SAM License, which you must accept to download the weights; read its terms before using them in a product. Your imagery has its own licence too.

## Next steps

-   Start with points, boxes and automatic masks in [Segment Anything on satellite and aerial imagery in Python](https://www.technicalgis.com/tutorials/segment-anything-satellite-imagery/).
-   Segment from text prompts, calibrated on one district and tested on another, in [Text-prompted Segment Anything on aerial imagery](https://www.technicalgis.com/tutorials/text-prompt-segment-anything-grounding-dino-sam2/).
-   Build your own training data and model in [training data for deep learning on satellite imagery](https://www.technicalgis.com/tutorials/satellite-imagery-training-data-python/) and [Train a U-Net for satellite image segmentation in PyTorch](https://www.technicalgis.com/tutorials/train-unet-satellite-segmentation-pytorch/).

## References and licences

-   Kirillov, A. et al. (2023). _Segment Anything_. arXiv:2304.02643. Code and weights: [Apache-2.0](https://github.com/facebookresearch/segment-anything/blob/main/LICENSE).
-   Ravi, N. et al. (2024). _SAM 2: Segment Anything in Images and Videos_. arXiv:2408.00714. Code and weights: [Apache-2.0](https://github.com/facebookresearch/sam2/blob/main/LICENSE).
-   Carion, N. et al. (2025). _SAM 3: Segment Anything with Concepts_. arXiv:2511.16719. SAM 3.1 followed in March 2026. Code and weights: [SAM License](https://github.com/facebookresearch/sam3/blob/main/LICENSE).
-   Liu, S. et al. (2023). _Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection_. arXiv:2303.05499. [Apache-2.0](https://github.com/IDEA-Research/GroundingDINO/blob/main/LICENSE).
-   Wu, Q. and Osco, L. (2023). _samgeo: A Python package for segmenting geospatial data with the Segment Anything Model (SAM)_. Journal of Open Source Software, 8(89), 5663.
-   Esri: [Text SAM](https://doc.arcgis.com/en/pretrained-models/latest/imagery/introduction-to-text-sam.htm) and [Segment Anything Model](https://doc.arcgis.com/en/pretrained-models/latest/imagery/introduction-to-segment-anything-model-sam-.htm) pretrained model documentation.
-   Figures: PDOK Luchtfoto 2025 25 cm, Beeldmateriaal Nederland, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); BGT, [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/); NAIP, US Department of Agriculture, Farm Service Agency, public domain; OpenStreetMap © OpenStreetMap contributors, [ODbL](https://opendatacommons.org/licenses/odbl/).

## Key terms

- **Segment Anything Model (SAM)**: A family of image segmentation models from Meta. Given a prompt such as a point, a box or, in SAM 3, a short phrase, it returns a mask of the object at that place. It was trained on more than a billion masks, so it outlines objects it has never been taught to recognise.
- **Prompt**: What you give SAM to say which object you want: one or more points (inside or outside the object), a box, a rough mask or, for SAM 3, a noun phrase or an example box.
- **Open-vocabulary detector**: An object detector that takes the class names as text at run time instead of having a fixed list. Grounding DINO is the one most often paired with SAM.
- **Mask and polygon**: SAM returns a mask: a grid of pixels marked in or out. To use it in GIS, the mask is converted to polygons using the image's geotransform, so the outline has real-world coordinates.
- **Pixel score and object score**: A pixel score asks how much of the object area is right; an object score asks how many individual objects were matched one to one, usually at IoU of 0.5 or more. The same result can score well on one and badly on the other.

## Data for this article

-   [PDOK (Publieke Dienstverlening Op de Kaart)](https://www.technicalgis.com/data-sources/netherlands/#src-pdok) · Luchtfoto 25 cm aerial photography used in the text-prompt tutorial
-   [BGT (Kadaster)](https://www.technicalgis.com/data-sources/netherlands/#src-kadaster-bgt) · the Dutch topographic register used as the reference

Licences and formats for each source are listed in the [GIS data sources catalogue](https://www.technicalgis.com/data-sources/).

---

Source: https://www.technicalgis.com/articles/segment-anything-for-gis/
