Algorithm for AI to identify optimal enclosure paths in 2D territory capture game

I’m working on an AI system for a territory capture game (like Paper.io or similar games where you draw lines to claim areas). The AI needs to be smart about finding the best spots to capture.

Basically I have a 2D grid where some cells are already owned (let’s call them occupied) and some are empty (unclaimed territory). The AI should figure out the most efficient way to draw a path that surrounds and captures the biggest chunk of empty space possible.

The main challenge is getting the AI to recognize good opportunities - like finding concave shapes or pockets of unclaimed area that would give maximum territory for minimum path length.

I’ve been thinking about using some kind of boundary detection method, but I’m not sure what approach would work best. Has anyone dealt with similar pathfinding problems where you need to optimize for area coverage rather than just shortest distance?

honestly, i’d start simple - raycast from occupied cells to find the biggest empty spaces. map those out, then use greedy pathfinding that targets areas near corners or map edges since they take fewer moves to close off. this approach worked pretty well for my clone.

Hit this exact problem a few years back when we were prototyping a similar game at work.

Treating it like region growing instead of pure pathfinding worked way better. I used morphological operations on the grid to find “capture seeds” - spots where you’ve already got good natural boundaries.

The trick was ranking potential enclosures by perimeter to area ratio. You want fat, round captures, not long skinny ones. Scan for clusters of empty cells, then calculate how much perimeter you’d need to close each cluster.

For path planning, dynamic programming worked great for building optimal enclosure paths. Start from your current position, flood fill to find all reachable boundary points, then work backwards to find which completion paths give the best area/cost ratio.

One thing that really helped - penalize paths that get too close to enemy territory. Nothing worse than getting your enclosure cut off mid-draw.

Preprocessing is crucial though. Run connected component analysis on empty regions first, then you only evaluate realistic capture targets instead of path-planning blindly.

Had to solve this exact problem for a competitive AI bot tournament last year. Distance transforms + watershed segmentation worked best for finding natural capture boundaries. Here’s the key insight: preprocess the grid to create a distance field from all occupied cells, then find local maxima. These spots are furthest from existing territory and usually mark the center of good capture opportunities. From there, use gradient descent along the distance field to find optimal enclosure paths that follow the terrain naturally. The game-changer was adding temporal analysis - I tracked how the territory map changes over multiple turns to predict opponent expansion. This lets you prioritize captures that block enemy growth paths. Major optimization: instead of evaluating every possible enclosure, I clustered similar capture opportunities and only computed detailed paths for the most promising clusters. Cut computation time by 70% with barely any impact on capture quality. My scoring function weighted area gain heavily but also factored in defensive value - sometimes capturing a smaller area that blocks multiple opponent expansion routes beats going for maximum territory.

Try alpha-beta pruning with potential field analysis. I built something similar for a uni AI course and got way better results by giving empty cells different values based on how close they were to boundaries - much better than just calculating area. The game-changer was using Voronoi diagrams to find cells equidistant from multiple owned territories. Those spots are usually your best capture opportunities. What really boosted performance was a two-phase approach: run edge detection on the territory map to spot promising regions, then use A* pathfinding with a custom heuristic that weighs both path length and potential area capture. Make sure your heuristic heavily favors moves that create bottlenecks or block opponent expansion. One thing that bit me - don’t let the AI chase huge enclosures when smaller guaranteed captures are sitting right there.

You should try a flood fill algorithm with convex hull detection. I built something similar for a mobile game prototype - preprocessing the grid to find ‘vulnerable’ boundaries worked really well. Just scan for occupied territory that forms concave patterns or has thin necks that could get cut off easily. I calculated potential area gain versus path cost for each possible enclosure move. Run flood fill from potential starting points and measure how much territory you’d capture by completing the enclosure. The hard part is figuring out where the optimal end point should be before you start drawing. What helped me was weighting moves that create enclosures near existing territory higher - they need shorter paths. Also add some look-ahead logic so you don’t get trapped in areas where you can’t complete a successful enclosure.