This website stores cookies on your computer. These cookies are used to collect information about how you interact with the website and allows us to remember you. We use this information in order to improve and customize your browsing experience and for analytics and metrics about visitors to this website. If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference not to be tracked.
Blog > Google > Search Engines

Search Engines

I want to make my site searchable. To do this we need spiders to crawl the website. So it's time to register with search engines. There are some variations depending on the source but the big three are Google, Bing and Yahoo.

Google

Let's start with google. If you have already set up the site for Google Analytics as described in this article, then you should already have a Google account. If you don't have one you will need to register for one. Once logged in with you google account you can go straight to Google's search console and enter in your domain.

Google Verification Dialogue Google Verification Dialogue

When you click on the Continue button google provides you with a code and it looks something like this: google-site-verification=aB123bcasaA-2babcBcC3211A. We need to add this to our AWS Route 53 Hosted Zone. This means we need to append to our infrastructure code from article URL to Somewhere.

Our starting point file structure:

tree ~/projects/website/infrastructure
~/projects/website/infrastructure
├── sources
│   ├── domain
│   │   ├── main.tf
│   │   └── variables.tf
│   └── store
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── terraform
    └── prod
        ├── domain
        │   └── terragrunt.hcl
        ├── store
        │   └── terragrunt.hcl
        └── terragrunt.hcl

7 directories, 8 files

Where the content of ~/projects/website/infrastructure/sources/domain/main.tf is

terraform {
  backend "s3" {}
}

provider "aws" {}

data "aws_caller_identity" "this" {}

data "aws_route53_zone" "this" {
  count = length(var.domain_names)
  name = "${element(var.domain_names, count.index)}."
}

resource "aws_route53_record" "this" {
  count = length(var.domain_names)

  zone_id = element(data.aws_route53_zone.this[*].zone_id, count.index)
  name    = element(var.domain_names, count.index)
  type    = "A"

  alias {
    name                   = element(var.domain_s3_bucket_website_domain, count.index)
    zone_id                = var.s3_bucket_dns_hosted_zone_id
    evaluate_target_health = false
  }
}

resource "aws_route53_record" "sub_domains" {
  count = length(var.domain_names)

  zone_id = element(data.aws_route53_zone.this[*].zone_id, count.index)
  name    = "www.${element(var.domain_names, count.index)}"
  type    = "A"

  alias {
    name                   = element(var.sub_domain_s3_bucket_website_domain, count.index)
    zone_id                = var.s3_bucket_dns_hosted_zone_id
    evaluate_target_health = false
  }
}

and the ~/projects/website/infrastructure/sources/domain/variables.tf has the content

variable "domain_names" {
  description = "A list of domain names"
  type        = list
}

variable "domain_s3_bucket_website_domain" {
  description = "A list dns names of the domain s3 buckets"
  type        = list
}

variable "s3_bucket_dns_hosted_zone_id" {
  description = "A hosted zone id of the s3 buckets"
}

variable "sub_domain_s3_bucket_website_domain" {
  description = "A list dns names of the subdomain s3 buckets"
  type        = list
}

Finally ~/projects/website/infrastructure/terraform/prod/domain/terragrunt.hcl has the content

include {
    path = "${find_in_parent_folders()}"
}

dependency "store" {
  config_path = "../store"
}

inputs = {
  domain_names                  = [
    "awebsitefromscratch.com",
    "awebsitefromscratch.co.uk"
  ]
  domain_s3_bucket_website_domain    = [
    dependency.store.outputs.domain_s3_bucket_website_domain,
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[0]
  ]
  sub_domain_s3_bucket_website_domain = [
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[1],
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[2]
  ]
  s3_bucket_dns_hosted_zone_id  = dependency.store.outputs.s3_bucket_dns_hosted_zone_id
}

What needs to change for Google

We need to add a new record set of type Text with a blank Name, no Alias, a default Time To Live and a value of the verification code that was provided by Google.

A new variable

We need a new variable, the value will need to be in the terragrunt file and it then needs to be referenced in the variables file.

Starting with ~/projects/website/infrastructure/terraform/prod/domain/terragrunt.hcl

include {
    path = "${find_in_parent_folders()}"
}

dependency "store" {
  config_path = "../store"
}

inputs = {
  domain_names                        = [
    "awebsitefromscratch.com",
    "awebsitefromscratch.co.uk"
  ]
  domain_s3_bucket_website_domain     = [
    dependency.store.outputs.domain_s3_bucket_website_domain,
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[0]
  ]
  sub_domain_s3_bucket_website_domain = [
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[1],
    dependency.store.outputs.sub_domain_s3_bucket_website_domain[2]
  ]
  s3_bucket_dns_hosted_zone_id        = dependency.store.outputs.s3_bucket_dns_hosted_zone_id
  google_verification_code            = "aB123bcasaA-2babcBcC3211A"
}

Then ~/projects/website/infrastructure/sources/domain/variables.tf

variable "domain_names" {
  description = "A list of domain names"
  type        = list
}

variable "domain_s3_bucket_website_domain" {
  description = "A list dns names of the domain s3 buckets"
  type        = list
}

variable "s3_bucket_dns_hosted_zone_id" {
  description = "A hosted zone id of the s3 buckets"
}

variable "sub_domain_s3_bucket_website_domain" {
  description = "A list dns names of the subdomain s3 buckets"
  type        = list
}

variable "google_verification_code" {
  description = "The verification code for Google search"
}

Last but by far the most important ~/projects/website/infrastructure/sources/domain/main.tf

terraform {
  backend "s3" {}
}

provider "aws" {}

data "aws_caller_identity" "this" {}

data "aws_route53_zone" "this" {
  count = length(var.domain_names)
  name = "${element(var.domain_names, count.index)}."
}

resource "aws_route53_record" "this" {
  count = length(var.domain_names)

  zone_id = element(data.aws_route53_zone.this[*].zone_id, count.index)
  name    = element(var.domain_names, count.index)
  type    = "A"

  alias {
    name                   = element(var.domain_s3_bucket_website_domain, count.index)
    zone_id                = var.s3_bucket_dns_hosted_zone_id
    evaluate_target_health = false
  }
}

resource "aws_route53_record" "sub_domains" {
  count = length(var.domain_names)

  zone_id = element(data.aws_route53_zone.this[*].zone_id, count.index)
  name    = "www.${element(var.domain_names, count.index)}"
  type    = "A"

  alias {
    name                   = element(var.sub_domain_s3_bucket_website_domain, count.index)
    zone_id                = var.s3_bucket_dns_hosted_zone_id
    evaluate_target_health = false
  }
}

resource "aws_route53_record" "google_txt" {
  name    = ""
  zone_id = element(data.aws_route53_zone.this[*].zone_id, 0)
  type    = "TXT"
  records = ["google-site-verification=${var.google_verification_code}"]
  ttl     = 300
}

Apply the code by running a terragrunt apply in the ~/projects/website/infrastructure/terraform/prod/domain directory

With the infrastructure built we just need to go back to the google dialog and click the verify button.

Google Verification Done, What Now?

Click the Burger Menu button in the top left corner and pick select you newly registered domain

Google Search Domain Property Home

Take a look at the top level reports about you site.

I was going to move on from here straight into Bing, but there is more to do in here than I expected. I will look at this for the next article.