How to do keyword research in 2026: a step-by-step guide

Keyword research used to mean pulling a volume number and a difficulty score and calling it a day. That workflow is broken now. Google’s AI Overviews sit on top of a growing share of informational queries, zero-click search has been documented for years, and a keyword with “10,000 searches a month” can send you almost no traffic if an AI summary answers the query before anyone clicks a blue link. I run several content sites out of Singapore, including this one, and the keyword research process I use today looks less like a spreadsheet of volume numbers and more like a map of which queries still reward a click.

This tutorial is for operators who are building or scaling a content site, not for someone doing a one-off blog post. If you’re running a single article a month, half of this will be overkill. If you’re trying to build topical authority across a niche site, or you’re managing a drip queue of articles the way I do for this site, you need a repeatable process that doesn’t require re-learning keyword tools every time.

By the end you’ll have a working method to go from a blank niche to a prioritized list of keyword clusters, each mapped to search intent and scored by whether it’s actually worth writing about in an AI Overview world. I’ll flag where the process breaks and what I do when it does.

what you need

a verified Google Search Console property for your domain (free, but the site needs to exist and be indexed first)
a Google Ads account for Keyword Planner access, free to create, no ad spend required to view keyword data
Google Trends for direction and seasonality, free
a paid keyword tool for volume and difficulty at scale, I use Ahrefs (Lite plan runs about $129/month); Semrush is a comparable alternative
a spreadsheet or Airtable base to track clusters, intent, and scores
python 3 if you want to script the clustering step (pandas and a sentence-embedding library, I use sentence-transformers)
a place to turn clusters into briefs: a CMS, a markdown repo, or a queue file if you’re running a drip-publish pipeline

step by step

1. audit what you already rank for

Before hunting new keywords, pull every query your domain already gets impressions for. In Search Console, go to Performance, set the date range to the last 12 months (or 16 months if the domain is older), and export the full Queries table, not just the top rows shown in the UI.

Expected output: a CSV or export with query, clicks, impressions, CTR, and average position for every query Google has recorded for your domain.

If it breaks: Search Console’s UI caps exports at 1,000 rows. Use the Search Console API instead, it has no such cap on the searchAnalytics.query method. A basic python script:

from googleapiclient.discovery import build
from google.oauth2 import service_account

creds = service_account.Credentials.from_service_account_file(
    "gsc-service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
service = build("searchconsole", "v1", credentials=creds)

request = {
    "startDate": "2025-07-01",
    "endDate": "2026-07-01",
    "dimensions": ["query"],
    "rowLimit": 25000,
}
response = service.searchanalytics().query(
    siteUrl="sc-domain:yoursite.com", body=request
).execute()

This existing-query data is your highest-signal input. Google already showed your page for these terms, which means intent match is proven, not guessed.

2. mine seed keywords from your own site and competitors

List every page on your site and note the core topic in one phrase. Then find three to five competitors ranking on page one for your existing top queries and run their domains through Ahrefs’ Site Explorer or Semrush’s Organic Research report to pull their top organic pages by traffic.

Expected output: a list of 50-200 seed phrases covering your niche plus adjacent topics competitors cover that you don’t.

If it breaks: if your niche is thin and competitors barely rank, widen the competitor set to include forums and communities (Reddit threads, niche subreddits) that show up in the SERP. Their thread titles are often keyword phrases people actually type.

Google’s autocomplete and “people also ask” boxes reflect real query patterns better than most paid tools, because they’re generated from live search behavior, not a third-party crawler’s estimate. You can pull autocomplete suggestions directly:

curl -s "https://suggestqueries.google.com/complete/search?client=firefox&q=keyword+research" | python3 -m json.tool

Run this against your seed list, then re-run it against the top suggestions it returns (autocomplete is recursive, so “keyword research tools” will surface a different set than “keyword research”).

Expected output: each seed expands into 5-10 long-tail variants, giving you several hundred candidate phrases from a few dozen seeds.

If it breaks: the endpoint is unofficial and Google can rate-limit or change the response format without notice. If it stops returning JSON, fall back to Ahrefs’ “Also rank for” and “Search suggestions” reports in Keywords Explorer, which pull from the same autocomplete data through an official API relationship.

4. pull volume and difficulty data

Feed your full candidate list into Keyword Planner (free, via a Google Ads account) or your paid tool. Keyword Planner’s volume ranges are famously bucketed and imprecise below certain thresholds, but it’s directionally reliable and it’s the tool Google itself uses to serve ads against these terms, per Google’s own Keyword Planner documentation.

Expected output: volume, competition, and (in Ahrefs/Semrush) a keyword difficulty score for every candidate.

If it breaks: if you don’t have ad spend history, Keyword Planner sometimes returns wide ranges like “1K-10K” instead of exact numbers. That’s normal, not a bug. Cross-reference with your paid tool’s estimate and trust the paid tool’s precision, use Keyword Planner mainly for coverage and Google’s own bucketing of related terms.

5. check the live SERP for AI Overviews and real intent

This is the step most operators skip and it’s the one that matters most in 2026. For your top 30-50 candidates by volume, search the term yourself, logged out, and note whether an AI Overview appears, what kind of pages rank below it (guides, tools, forums, product pages), and whether the intent is informational, commercial, or transactional.

Expected output: a column added to your spreadsheet marking each keyword AIO-present or clean, plus a one-word intent label.

If it breaks: AI Overview presence varies by location and query phrasing, and it’s not exposed through any keyword tool’s API reliably yet. There’s no shortcut here, you have to look at the actual SERP. If you’re checking hundreds of keywords, prioritize the ones with the highest volume first and sample the rest.

6. cluster keywords by topic and intent

Group your scored list into clusters that a single well-built page (or a small pillar-plus-supporting-pages structure) could serve. For small lists, do this manually in the spreadsheet by sorting alphabetically and eyeballing overlap. For larger lists, embed each phrase and cluster by cosine similarity:

from sentence_transformers import SentenceTransformer
from sklearn.cluster import AgglomerativeClustering
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
keywords = ["keyword research tools", "best keyword research tool", "how to find keywords"]
embeddings = model.encode(keywords)

clustering = AgglomerativeClustering(
    n_clusters=None, distance_threshold=0.4, metric="cosine", linkage="average"
).fit(embeddings)

Expected output: clusters of 3-15 related phrases, each representing one content asset, not one keyword per article.

If it breaks: over-clustering (everything ends up in one giant group) usually means your distance_threshold is too high. Lower it in increments of 0.05 and re-run until cluster sizes look sane for your niche.

7. score clusters by opportunity vs effort

For each cluster, combine total volume, average difficulty, AI Overview presence, and how well it fits your site’s existing topical authority. I use a simple weighted score: volume rank minus difficulty rank minus an AIO penalty, plus a bonus if the cluster reinforces a topic I already rank for from step 1.

Expected output: a ranked list of clusters, top to bottom, ready to schedule.

If it breaks: if everything scores similarly and nothing stands out, your seed list was probably too narrow in step 2. Go back and widen the competitor set.

8. turn the top clusters into content briefs

For each prioritized cluster, write a one-page brief: target intent, the questions to answer (pulled straight from your “people also ask” data), competitors to beat, and internal links to existing pages on your site. Feed this into whatever publishing queue or CMS you run.

Expected output: a backlog of briefs ready for writing, not just a list of keywords.

If it breaks: briefs that are too thin get generic articles that don’t outrank anyone. If your brief doesn’t include at least three specific sub-questions pulled from real SERP data, go back to step 3.

common pitfalls

chasing volume over intent. A 5,000-search keyword with heavy AI Overview coverage and mostly forum results ranking below it will send you less traffic than a 500-search keyword with a clean SERP and buyer intent.
treating Keyword Planner’s ranges as exact numbers. They’re buckets, not precise counts. Don’t build a whole content plan around a single tool’s guess.
skipping the manual SERP check. Every automated tool tells you what a keyword used to look like. Only a live search tells you what it looks like today, and AI Overviews change SERP structure fast.
one keyword per article. Google has consolidated rankings around topical pages for years now. A cluster of 8 related phrases usually deserves one page with strong subheadings, not eight thin pages competing with each other.
no re-audit cadence. Keyword research done once goes stale. Search behavior shifts, especially around anything AI-related, and a list from even six months ago can misjudge which SERPs are AIO-heavy now.

scaling this

At 10 articles, do everything manually. A spreadsheet, an afternoon on Keyword Planner, and your own judgment on intent is enough, and speed matters more than process at this size.

At 100 articles, you need the clustering step scripted, because manual grouping starts producing overlapping or duplicate briefs. This is also the point where I stop trusting my own SERP checks for every keyword and start sampling: check AI Overview presence on the top 20% of clusters by volume, and assume similar coverage for the long tail unless something looks off.

At 1,000+ articles, typically across a network of niche sites rather than one domain, the bottleneck moves from keyword discovery to prioritization and deduplication across sites. You’ll want a shared database of scored clusters rather than per-site spreadsheets, so you’re not accidentally building the same cluster on two different domains. This is also where a Search Console API pull becomes mandatory rather than optional, since manual export limits make it impossible to audit that many properties by hand.

where to go next

Once you’ve got scored clusters, the next problem is proving the content earns Google’s trust, not just matching a keyword. Read E-E-A-T explained: what Google actually rewards for how experience and expertise signals factor into whether a page ranks at all. Then, before you start writing individual briefs, map your clusters onto a full site structure with how to build a topical authority content plan, which covers how to sequence pillar and supporting content so clusters reinforce each other instead of competing.

If you’re evaluating AI-assisted clustering or brief-writing tools beyond what I’ve covered here, I track tool-specific breakdowns over at aitoolgazette.com/blog. For more tutorials like this one, browse the full archive at the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-15.