Error when crawlers load page

[ivanjolic95]

When accessed from browser everything works normally, but for some reason when crawlers (Google, Baidu) try to scan the page there's a 404. Logs look like this:

Started GET "/" for 104.193.88.245 at 2016-06-06 13:31:59 -0400
F, [2016-06-06T13:31:59.850052 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1]   
F, [2016-06-06T13:31:59.851321 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] ActionController::RoutingError (No route matches [GET] "/"):
F, [2016-06-06T13:31:59.851933 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1]   
F, [2016-06-06T13:31:59.852024 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/actionpack/lib/action_dispatch/middleware/debug_exceptions.rb:53:in `call'
F, [2016-06-06T13:31:59.852074 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/actionpack/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
F, [2016-06-06T13:31:59.852140 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/railties/lib/rails/rack/logger.rb:36:in `call_app'
F, [2016-06-06T13:31:59.852339 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/railties/lib/rails/rack/logger.rb:24:in `block in call'
F, [2016-06-06T13:31:59.852574 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/activesupport/lib/active_support/tagged_logging.rb:70:in `block in tagged'
F, [2016-06-06T13:31:59.852775 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/activesupport/lib/active_support/tagged_logging.rb:26:in `tagged'
F, [2016-06-06T13:31:59.855699 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/activesupport/lib/active_support/tagged_logging.rb:70:in `tagged'
F, [2016-06-06T13:31:59.855828 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/railties/lib/rails/rack/logger.rb:24:in `call'
F, [2016-06-06T13:31:59.855892 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] request_store (1.3.1) lib/request_store/middleware.rb:9:in `call'
F, [2016-06-06T13:31:59.856627 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/actionpack/lib/action_dispatch/middleware/request_id.rb:24:in `call'
F, [2016-06-06T13:31:59.856708 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] rack (2.0.0.alpha) lib/rack/method_override.rb:22:in `call'
F, [2016-06-06T13:31:59.857076 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] rack (2.0.0.alpha) lib/rack/runtime.rb:22:in `call'
F, [2016-06-06T13:31:59.857132 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/activesupport/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
F, [2016-06-06T13:31:59.857299 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/actionpack/lib/action_dispatch/middleware/executor.rb:12:in `call'
F, [2016-06-06T13:31:59.857371 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] rack (2.0.0.alpha) lib/rack/sendfile.rb:111:in `call'
F, [2016-06-06T13:31:59.857424 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] /home/tourgent/.rvm/gems/ruby-2.2.3/bundler/gems/rails-716571463745/railties/lib/rails/engine.rb:522:in `call'
F, [2016-06-06T13:31:59.857473 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/configuration.rb:224:in `call'
F, [2016-06-06T13:31:59.857529 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/server.rb:569:in `handle_request'
F, [2016-06-06T13:31:59.857580 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/server.rb:406:in `process_client'
F, [2016-06-06T13:31:59.857615 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/server.rb:271:in `block in run'
F, [2016-06-06T13:31:59.857665 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/thread_pool.rb:114:in `call'
F, [2016-06-06T13:31:59.857711 #17158] FATAL -- : [cee72b94-a7fc-47d1-9b5a-7484ddb25fd1] puma (3.4.0) lib/puma/thread_pool.rb:114:in `block in spawn_thread'

I'm not really sure what causes the bug. Thanks for help.

Which operating system and version is the project developed on?

Linux Mint 17

Which version of ruby is the project developed on?

2.2.3-p173

What is the rails version?

5.0.0.beta3

What is the ember-cli-rails version (from Gemfile)?

0.7.1

Is your application server multi-threaded (such as puma and unicorn) or is
it multi-process (such as thin and webrick)?

puma

What are the contents of config/initializers/ember.rb?

FRONTEND_PATH_BASE = Rails.root.join("frontend")

CLIENTS_PATH = FRONTEND_PATH_BASE.join("clients").to_s
TOUR_REQUESTS_PATH = FRONTEND_PATH_BASE.join("tour-requests").to_s

EmberCli.configure do |c|
  c.app :clients, path: CLIENTS_PATH
  c.app :tour_requests, path: TOUR_REQUESTS_PATH
end

What are the contents of the Rails' view that renders the Ember application?

default

How are the EmberCLI-related routes defined?

  mount_ember_app :tour_requests, to: '/tour-request'
  mount_ember_app :clients, to: '/'

How is the application deployed?

It's on DigitalOcean. (nginx)

[seanpdoyle]

@ivanjolic95 thanks for filing this issue.

The mount_ember_application routing helper has an HTML contstraint that will only accept requests with an HTML format.

Do you have access to the metadata or headers of the requests the crawlers are making?

[ivanjolic95]

Unfortunately I don't have the access, is there any way to make it work on the server side?

[seanpdoyle]

The idea with the constraint is to catch all HTML requests and serve them the Ember app, so that other request types (for example, JSON API requests) could be served from the same Rails application.

Are these crawlers not requesting HTML responses?

[ivanjolic95]

I can't figure out how it works internally, I guess they send a request that's obviously not an HTML request. It seems to me that the only way to see if it's from a bot is to look into user agent token/string in the request as stated here.

A solution would be to let requests with those user agent tokens get the ember app route, right?

[ivanjolic95]

Okay, so this seems to have worked for me ivanjolic95@ebc8ef3. I've needed a quick fix so thats what I came up with. If you're interested, I could refactor it a little bit to include other bots too and make a pull request. What do you think?

[seanpdoyle]

@ivanjolic95 thanks, we'd love a PR!

[ahlivehr]

I'm having the same issue. Whenever the Accept header is set to */*, which is the case for googlebot, the page returns a 404. I tried looking at your solution @ivanjolic95, but it looks like the fix was added to an outdated forked repo. Does anyone have any suggestions for how to fix this in the current state of the repo?