No such file to load -- lib/discourse.rb (LoadError)

[ioquatix]

koyoko% RAILS_ENV=profile ruby seed_db_data.rb
Traceback (most recent call last):
	21: from seed_db_data.rb:65:in `<main>'
	20: from /home/samuel/.rvm/rubies/ruby-2.6.1/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	19: from /home/samuel/.rvm/rubies/ruby-2.6.1/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	18: from /home/samuel/Documents/socketry/rails_ruby_bench/work/discourse/config/environment.rb:4:in `<top (required)>'
	17: from /home/samuel/.rvm/rubies/ruby-2.6.1/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	16: from /home/samuel/.rvm/rubies/ruby-2.6.1/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	15: from /home/samuel/Documents/socketry/rails_ruby_bench/work/discourse/config/application.rb:52:in `<top (required)>'
	14: from /home/samuel/Documents/socketry/rails_ruby_bench/work/discourse/config/application.rb:53:in `<module:Discourse>'
	13: from /home/samuel/Documents/socketry/rails_ruby_bench/work/discourse/config/application.rb:69:in `<class:Application>'
	12: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:246:in `require_dependency'
	11: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:334:in `depend_on'
	10: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:356:in `require_or_load'
	 9: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:37:in `load_interlock'
	 8: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies/interlock.rb:13:in `loading'
	 7: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
	 6: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
	 5: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:37:in `block in load_interlock'
	 4: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:373:in `block in require_or_load'
	 3: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:475:in `load_file'
	 2: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:661:in `new_constants_in'
	 1: from /home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:476:in `block in load_file'
/home/samuel/.rvm/gems/ruby-2.6.1/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:476:in `load': No such file to load -- lib/discourse.rb (LoadError)

I had to do bundle update to get anything to work in the root directory but get to this step and it fails.

[noahgibbs]

Yeah, there's a lot to get to work :-/ Question: if I could set up an EC2 instance for you, would that be an easier way to run the tests you have in mind?

[ioquatix]

I just wanted to throw falcon at it and see how it compared but I also want to use async-postgres so might require some hacking.

[noahgibbs]

That's fair. I'll say that by far the best supported bits here are the Packer build for an EC2 image. Just changing out for Falcon should be straightforward, in that I've occasionally run with another app server and it's not usually difficult. No clue about async-postgres - not sure how similar the interface is. But if it's at all useful for me to set up an instance for you with the current config to hack on, either from an existing built AMI or one you've built, please let me know.

I'll look at your config problem too, but it's very likely to boil down to "maintaining a locally installed copy of Discourse and keeping it running is very hard, and then you have some extra requirements on top of that." :-(

(The Discourse folks do this with Docker. I, alas, use this for benchmarking.)

[ioquatix]

async-postgres is a monkey patch on top of pg so in theory it's completely compatible, it's just a matter of loading it (i.e. require 'async/postgres' if defined? Falcon).

[ioquatix]

Rather than using docker, for my own benchmark, I used systemd-machinectl and pacman: https://github.com/socketry/falcon-benchmark

While it is a bit more platform specific than I'd like, it's super simple to deploy and test on a suitable Linux box and it runs on the bare metal where possible.

[ioquatix]

Sorry, I didn't really make any useful statement.

What about this: it sounds like you already have things set up on your end, do you just want to try it with falcon and we can discuss the results here? You can try adding that one line above - it will break migrations (rake tasks) but it should be fine when running the actual web app.

[noahgibbs]

Sure, I'll try that out and let you know the results - will likely take about 18 hours since I'm on UK time right now, and so I'll be sleeping before the next workday :-)

[ioquatix]

There is no hurry, just take your time :)

[noahgibbs]

Here's what I did:

• Started up an AMI
• Switched to Ruby 2.6.0
• Switched out puma for falcon (v. 0.31.0) in discourse/Gemfile, added async-postgres (v. 0.1.0)
• Falcon params: -b http://localhost:9292 --hybrid --forks 10 -- threads 6 to roughly mimic Puma's settings (also 10/6)
• Added a config/initializer/300-async-pg.rb with the "require" line for async-postgres
• Some edits to start.rb - removed some Puma-specific stuff like mem checking, changed startup/shutdown code
• Added config.allow_concurrency = true to config/environments/profile.rb
• Set up a little script to run 30 batches of: bundle exec ./start.rb -i 10000 -w 1000 -s 0 --no-warm-start -o data/

This seems nearly right. I can manually run "falcon serve -b http://localhost:9292" and have it serve up files using curl. But if I run "falcon serve --hybrid --forks 10 --threads 6" in the same directory it seems to die continually:

ubuntu@ip-172-31-49-58:~/rails_ruby_bench/work/discourse$ RAILS_ENV=profile bundle exec falcon serve -b http://localhost:9292 --hybrid --forks 10 --threads 6
/home/ubuntu/.rvm/gems/ruby-2.6.0/gems/activesupport-4.2.8/lib/active_support/core_ext/object/duplicable.rb:111: warning: BigDecimal.new is deprecated; use BigDecimal() method instead.
/home/ubuntu/.rvm/gems/ruby-2.6.0/gems/onebox-1.8.6/lib/onebox/preview.rb:5: warning: constant Net::HTTPServerException is deprecated
 4.04s: #<Falcon::Endpoint http://localhost:9292/>
      | Falcon taking flight! Using Async::Container::Hybrid {:count=>8, :forks=>10, :threads=>6}
      | - To terminate: Ctrl-C or kill 10400
 4.12s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10413 exit 0
 4.12s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10412 exit 0
 4.13s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10416 exit 0
 4.13s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10417 exit 0
 4.14s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10419 exit 0
 4.15s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10418 exit 0
 4.16s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10421 exit 0
 4.16s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10414 exit 0
 4.17s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10415 exit 0
 4.19s: #<Async::Container::Hybrid:0x00000000098629d8>
      | Falcon Server pid 10420 exit 0
(Many more of these follow)

They don't seem to leave anything in the Rails log or the other obvious places I've tried looking.

[noahgibbs]

It looks like "--forks 10" is the problem - just "--hybrid" or "--hybrid --threads 6" both run without that problem.