Segment Anything for GIS: SAM, SAM 2 and SAM 3 on aerial and satellite imagery

What Meta's Segment Anything models are, how to prompt them with points, boxes and text, the tools for using them in Python, ArcGIS Pro and QGIS, how to turn masks into GIS features, and how to measure whether the results are good enough, with measured results from two tutorials.

Three panels of the same corner of Utrecht: boxes from a text prompt over blocks of flats, the SAM 2.1 masks inside the kept boxes, and the resulting polygons following the national register's building outlines

Key takeaways

  1. Segment Anything draws outlines; it does not name what it outlines. Points, boxes or a text prompt tell it where to look. SAM and SAM 2 need a separate detector, such as Grounding DINO, for text; SAM 3 (November 2025) reads short text prompts by itself.
  2. SAM, SAM 2 and SAM 2.1 are Apache-2.0 and run on a CPU. SAM 3 and 3.1 are released under Meta's own SAM License, with gated weights, and in practice need an NVIDIA GPU.
  3. In our tests, boxes from an existing map gave the best outlines: a median IoU of 0.71 against OpenStreetMap houses, 145 of 146 above 0.5. A text prompt, tuned on one Utrecht district, found building area on another at pixel F1 0.70.
  4. Measure the results against a reference and choose the score that matches the question. On the same polygons, building area scored F1 0.70 but individual buildings only 0.08, because terraces came out as one roof and sheds were missed.

Meta’s Segment Anything models changed what it takes to outline things in imagery. Before 2023, extracting buildings, trees or ponds from aerial photographs meant training a model on thousands of labelled examples, or digitising by hand. SAM outlines almost any object in a fraction of a second once you tell it where to look, with no training at all.

That makes it very useful for GIS work, but it is easy to overestimate. SAM finds edges well, but it does not know what a building is. It works on 1024-pixel images, not on maps. And its results need checking like any other classification. This article explains what the models are, how to prompt them, which tools bring them into Python, ArcGIS Pro and QGIS, and how to measure the results. The figures come from two tutorials we ran end to end:

The models: SAM, SAM 2 and SAM 3

Model Released Works on Prompts Licence Runs on a CPU?
SAM April 2023 images points, boxes, masks Apache-2.0 yes, with ViT-B (94 M parameters)
SAM 2 and 2.1 July and September 2024 images and video points, boxes, masks Apache-2.0 yes; Hiera Large is 217 M parameters
SAM 3 November 2025 images and video text, example boxes, points, boxes, masks SAM License (gated) not in practice; 848 M parameters
SAM 3.1 March 2026 images and video as SAM 3 SAM License (gated) as SAM 3

SAM (Kirillov et al., 2023) has a heavy image encoder, which turns a 1024 × 1024 image into an embedding once, and a light decoder, which turns each prompt into a mask in about 30 milliseconds. It was trained on SA-1B: 11 million photographs with 1.1 billion masks. On a CPU, the smallest version, ViT-B, encoded a tile in about 4 seconds in our first tutorial.

SAM 2 (Ravi et al., 2024) replaced the encoder with a faster one, Hiera, and added a memory that follows an object from one video frame to the next. The September 2024 checkpoints, SAM 2.1, are the ones to use. For GIS, the video memory matters when you treat a time series of images as frames: an object prompted once in the first year can be followed through the rest.

SAM 3 (November 2025) adds what the earlier models lacked: it accepts a short noun phrase, such as “solar panel”, or an example box, and returns every matching object in the image. Meta calls this promptable concept segmentation. SAM 3.1 (March 2026) is a drop-in update that tracks many objects in video faster. Both are released under Meta’s SAM License rather than Apache-2.0. The weights are gated: you request access on Hugging Face and accept the licence before you can download them.

How to prompt it

SAM always needs to be told where to look. The options, from the most control to the least:

  • Boxes. A box around one object is the most reliable prompt, because it also says how big the object is. Boxes can come from an existing map, a detector or a person dragging a rectangle. In our first tutorial, boxes drawn around OpenStreetMap footprints gave a median IoU of 0.71, and 145 of 146 houses scored above 0.5.
  • Points. A click inside the object, optionally with clicks outside it to exclude areas. One point is ambiguous (a chimney, a roof plane or the whole house?), so SAM returns three candidate masks with its own quality estimate for each. Points suit interactive digitising.
  • Automatic. SAM places a grid of points over the image and keeps the best, non-overlapping masks. It outlines everything, roofs, trees, car parks and playing fields, without saying which is which. You then need rules or a classifier to keep the objects you want. In our first tutorial, automatic masks outlined 73% of the OpenStreetMap buildings, but after rules on area, shape and NDVI to separate buildings from everything else, 55% remained.
  • Text. With SAM and SAM 2, a text prompt goes to an open-vocabulary detector, usually Grounding DINO, which returns boxes, and the boxes go to SAM. With SAM 3, the text goes to the model directly.
Left: a zoomed area of houses around two cul-de-sacs, with dashed yellow OpenStreetMap outlines and solid red SAM outlines that follow each roof closely. Right: a histogram of IoU between SAM masks and OpenStreetMap footprints for 146 buildings, clustered between 0.6 and 0.85 with a median of 0.71.
Box prompts from OpenStreetMap footprints on NAIP imagery of Ames, Iowa: SAM's outlines (red) against the footprints (dashed yellow). From our samgeo tutorial.

Text prompts need calibrating

Text prompts are the most convenient, and the least predictable. In the text-prompt tutorial, Grounding DINO’s scores on aerial imagery were low: nine boxes in ten scored below 0.30, under the 0.35–0.4 defaults suggested for everyday photographs. At 0.35, it found no trees at all. The wording mattered too. For buildings, “house.” scored pixel F1 0.75, “building.” 0.74 and “roof.” 0.67. For trees, “a tree.” beat “tree.”. Cutting the image into smaller tiles, 512 pixels rather than 1024, found 2.6 times as many cars, because Grounding DINO enlarges a small tile and the cars become bigger.

The safe way to use a text prompt is to choose the wording, tile size and threshold on an area where you have a reference, and then check them on a second area before running everything else. The settings we chose on Kanaleneiland in Utrecht gave a building F1 of 0.70 on Parkwijk, 4 km away, the same as the best Parkwijk could reach with its own settings.

Two districts of Utrecht with every polygon from four text prompts: buildings in orange, tree crowns in green, water in blue and a few cars in yellow. In Parkwijk, a paved square beside the railway is also shaded blue.
Four text prompts, with settings chosen on Kanaleneiland (left) and tested on Parkwijk (right). The blue square beside the railway in Parkwijk is paving, not water. From our text-prompt tutorial.

The tools

Python

  • segment-geospatial (samgeo), by Qiusheng Wu, MIT licence. It wraps SAM, SAM 2 and SAM 3 for geospatial data: it reads GeoTIFFs and web map tiles, accepts prompts in map coordinates, writes masks as GeoTIFF or vector, and includes LangSAM for text prompts with Grounding DINO. SAM 3 support is installed with the segment-geospatial[samgeo3] extra. This is the quickest start.
  • Hugging Face transformers includes Grounding DINO, SAM, SAM 2 and SAM 3 as standard classes. You handle tiling and georeferencing yourself, but you control every step and can run on a CPU without compiling anything. Our text-prompt tutorial uses this route.
  • Meta’s own repositories (sam2, sam3) have the reference code, the video predictors and the newest features first.

ArcGIS Pro

Esri publishes two ready-made deep learning packages on ArcGIS Living Atlas, used with the Detect Objects Using Deep Learning tool in the Image Analyst extension:

Esri recommends a GPU with at least 8 GB of memory for both, and they take 8-bit, 3-band imagery. Both packages need the ArcGIS deep learning libraries installed into Pro’s Python environment.

QGIS

  • GeoAI plugin (opengeos, MIT licence), by the samgeo author, brings samgeo’s SAM, SAM 2 and SAM 3 tools and other GeoAI models into QGIS. It is the successor its author now recommends over the earlier SamGeo plugin.
  • Geo-SAM (MIT licence) encodes an image once, ahead of time, and then lets you digitise by clicking points or dragging boxes, with each mask appearing almost at once. It handles points and boxes only.
  • GeoOSAM (GPL v2) supports SAM 2.1 and SAM 3, including free SAM 3 text prompts on the visible map extent; automatic tiling over larger areas is a paid feature.

Interactive plugins suit digitising, where a person checks every outline as it appears. For unattended extraction over a large area, a script is easier to reproduce and to measure.

From masks to GIS features

A mask is a grid of pixels, so turning it into a GIS feature follows the same steps as any raster-to-vector conversion:

  1. Keep the geotransform of each tile. SAM sees a plain 1024 × 1024 image. If you cut a GeoTIFF into tiles, each tile’s origin must be carried through, or every polygon will land at the corner of the map.
  2. Polygonise with rasterio.features.shapes (or Raster to Polygon in ArcGIS, Polygonize in QGIS), drop specks below a minimum area, and simplify by about half a pixel to remove the staircase edges.
  3. Handle tile edges. Objects cut by a tile edge come out in pieces. Overlap the tiles and either keep each object only in the tile whose centre region holds it, or merge overlapping polygons of the same class afterwards.
  4. Keep the scores. Store the detector’s score and SAM’s own quality estimate with each polygon, so you can filter or review the doubtful ones later.

Our raster to vector tutorial covers the polygonising and cleaning in more depth.

How to check the results

A SAM result looks convincing on screen, which makes measuring it more important, not less. Three things decide what a score means.

The reference. OpenStreetMap is free everywhere but incomplete; national registers such as the Dutch BGT or Ordnance Survey MasterMap are complete but follow their own rules. The BGT draws buildings at the walls, while the camera sees the roof with its overhang; it records each terraced house separately, but from above a terrace is one roof. Utrecht registers every street and park tree, but not trees in private gardens, so a correct garden tree counts as a false positive.

The score. Choose the measure that answers your question:

On the Parkwijk test site Score
Building area, pixel by pixel F1 0.70 (IoU 0.54)
Individual BGT buildings, matched one to one at IoU ≥ 0.5 F1 0.08
Roof blocks (touching buildings merged), matched one to one F1 0.35
Registered trees inside a predicted crown 67% of 533
Water area, pixel by pixel F1 0.40

All five rows describe the same run. If you want to know how much of a district is roofed, it did well. If you want one polygon per address, it did badly.

Calibration and test. Choosing the prompt and threshold on the same area you score inflates the result. Score on a second area that played no part in the choice.

Where it fails

Both tutorials found the same kinds of failure:

  • Small objects. In Parkwijk, none of the 200 buildings under 20 m² was found, and across all 440 buildings under 150 m² the text prompt matched 7. At 25 cm, a 10 m² shed is about 12 × 12 pixels. Small tiles help, but only so far.
  • Joined objects. A terrace of eight houses, or a school with several roof levels, comes out as one mask or as several pieces that match nothing. In Ames, SAM split a school’s flat roof at its parapets. In Utrecht, Grounding DINO drew one box around a whole terrace.
  • Look-alikes. Text prompts follow appearance, not meaning. “water.” matched three flat roofs and a paved station square in Parkwijk, and dark shadow beside blocks of flats in Kanaleneiland. Pixel F1 for water was 0.40 on the test site.
  • Trees over buildings. Houses partly under trees gave masks that included the canopy, or were dropped by a vegetation rule.
  • Anything not like a photograph. SAM was trained on ordinary colour photographs, and both of our tests used RGB aerial imagery. Radar, elevation and single-band data look very different; render them as an image with clear edges, such as a hillshade, before prompting, and measure the result against a reference.
Parkwijk with polygons coloured by match: large apartment blocks in green, whole terraces of houses in orange with thin red outlines of the individual houses inside, and small red squares of garden sheds that have no polygon.
Parkwijk building errors from the text prompt 'house.': green polygons match one BGT building, orange ones match none, and red outlines are BGT buildings not matched. The terraces are found as whole rows.

When to use it, and when not

Segment Anything is a good choice when:

  • you have boxes or points already, from a map, a detector or a person, and want accurate outlines;
  • you are digitising interactively and want each click to give a clean polygon;
  • you have no labelled training data and need a first result to check and correct.

Other tools are often better when:

  • you need one feature per address or per tree: start from existing footprints or train a model on your own labels, as in our U-Net tutorial;
  • you have lidar: building footprints and heights from lidar are more complete than anything traced from a photograph;
  • you need land cover for every pixel rather than objects: a classifier on Sentinel-2 answers that directly;
  • you need to map water: a spectral index on near-infrared bands separates water from paving and shadow, which a text prompt on RGB did not.

Frequently asked questions

What is Segment Anything in GIS?

It is Meta’s family of image segmentation models, SAM, SAM 2 and SAM 3, used on aerial, drone and satellite imagery to outline features such as buildings, trees, fields and water. You prompt it with points, boxes or text, and convert the resulting masks to polygons with the image’s geotransform.

Which Segment Anything model should I use?

For box and point prompts on a CPU, SAM 2.1 is the best choice: it is Apache-2.0, faster than the original SAM and more accurate. For text prompts without a GPU, pair SAM 2.1 with Grounding DINO. If you have an NVIDIA GPU and accept the SAM License, SAM 3 reads text prompts directly and returns every matching object.

Does ArcGIS Pro support Segment Anything?

Yes. Esri publishes a Segment Anything Model package and a Text SAM package, which pairs Grounding DINO with SAM, on ArcGIS Living Atlas. Both run with the Detect Objects Using Deep Learning tool in the Image Analyst extension, and Esri recommends a GPU with at least 8 GB of memory.

Is there a QGIS plugin for Segment Anything?

Several. The GeoAI plugin brings samgeo’s SAM, SAM 2 and SAM 3 tools into QGIS; Geo-SAM offers fast interactive digitising with points and boxes; and GeoOSAM supports SAM 2.1 and SAM 3, including text prompts.

How accurate is Segment Anything on aerial imagery?

It depends on the prompt and on what you measure. With boxes from OpenStreetMap footprints, SAM’s outlines of houses had a median IoU of 0.71. With the text prompt “house.”, on a district not used for tuning, building area scored pixel F1 0.70, but only 22 of its 464 individual buildings were matched one to one, because terraces came out as one roof and sheds were missed.

Can I use Segment Anything commercially?

SAM, SAM 2 and SAM 2.1 are released under Apache-2.0, which allows commercial use. Grounding DINO is also Apache-2.0. SAM 3 and SAM 3.1 use Meta’s SAM License, which you must accept to download the weights; read its terms before using them in a product. Your imagery has its own licence too.

Next steps

References and licences

  • Kirillov, A. et al. (2023). Segment Anything. arXiv:2304.02643. Code and weights: Apache-2.0.
  • Ravi, N. et al. (2024). SAM 2: Segment Anything in Images and Videos. arXiv:2408.00714. Code and weights: Apache-2.0.
  • Carion, N. et al. (2025). SAM 3: Segment Anything with Concepts. arXiv:2511.16719. SAM 3.1 followed in March 2026. Code and weights: SAM License.
  • Liu, S. et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv:2303.05499. Apache-2.0.
  • Wu, Q. and Osco, L. (2023). samgeo: A Python package for segmenting geospatial data with the Segment Anything Model (SAM). Journal of Open Source Software, 8(89), 5663.
  • Esri: Text SAM and Segment Anything Model pretrained model documentation.
  • Figures: PDOK Luchtfoto 2025 25 cm, Beeldmateriaal Nederland, CC BY 4.0; BGT, CC0 1.0; NAIP, US Department of Agriculture, Farm Service Agency, public domain; OpenStreetMap © OpenStreetMap contributors, ODbL.

Key terms

Segment Anything Model (SAM)
A family of image segmentation models from Meta. Given a prompt such as a point, a box or, in SAM 3, a short phrase, it returns a mask of the object at that place. It was trained on more than a billion masks, so it outlines objects it has never been taught to recognise.
Prompt
What you give SAM to say which object you want: one or more points (inside or outside the object), a box, a rough mask or, for SAM 3, a noun phrase or an example box.
Open-vocabulary detector
An object detector that takes the class names as text at run time instead of having a fixed list. Grounding DINO is the one most often paired with SAM.
Mask and polygon
SAM returns a mask: a grid of pixels marked in or out. To use it in GIS, the mask is converted to polygons using the image's geotransform, so the outline has real-world coordinates.
Pixel score and object score
A pixel score asks how much of the object area is right; an object score asks how many individual objects were matched one to one, usually at IoU of 0.5 or more. The same result can score well on one and badly on the other.
Two districts of Utrecht in summer aerial photography, with buildings shaded orange, tree crowns green, water blue and cars yellow, each found from a one-word text prompt

Tutorial Python

Text-prompted Segment Anything on aerial imagery: Grounding DINO and SAM 2.1 in Python

Segment buildings, trees, water and cars from a text prompt with Grounding DINO and SAM 2.1 in Python on a CPU, the open equivalent of ArcGIS Pro's Text SAM. Choose the prompt wording, tile size and threshold on one Utrecht district, then measure accuracy on another against the Dutch BGT register.

Advanced 17 min read

An aerial image of a housing estate covered in 306 coloured Segment Anything masks beside the same image without masks

Tutorial Python

Segment Anything on satellite and aerial imagery in Python: a samgeo tutorial

Run Meta's Segment Anything Model on high-resolution aerial imagery with segment-geospatial (samgeo), on a CPU. Generate masks automatically, vectorise them to a GeoPackage, filter them by area, shape and NDVI to find buildings, and prompt SAM with points and boxes, checked against OpenStreetMap.

Intermediate 14 min read

Two habitat maps of the Dark Peak side by side. Left: the rule-based map from Sentinel-2, with purple blanket bog and lilac heath on the Kinder and Bleaklow plateaux, orange bracken on the valley sides, green grassland and woodland in the valleys and blue reservoirs in the Derwent valley. Right: Living England 2022-23 for the same area, with smoother patches of the same classes.

Tutorial Python Part 1 of 5

Habitat mapping with Sentinel-2 in Python: spectral index rules and an honest accuracy assessment

Map blanket bog, heath, bracken, grassland, woodland, water and built-up land in the Peak District from 2023 Sentinel-2 seasonal composites with an eight-rule spectral index classifier, then assess it properly: a confusion matrix, user's and producer's accuracy, kappa and its caveats, block-bootstrap confidence intervals, and why a random train/test split overstates accuracy.

Beginner 19 min read