Moving to a Rails Engine
At work we have a primary Rails application (ye ole monolith) and several microservices that handle our business, which is an email newsletter provider and delivery system. As you can probably imagine, email newsletters are sent to, well, emails! When a client adds new emails (read: people) to the system, we define them as members.
Simple Models, Complex Actions
The basic structure of our models, as it relates to new members, is as follows (pretty easy and straight forward):
class Member < ApplicationRecord
belongs_to :client
# there are other relationships but they're not really relevant here
end
class Client < ApplicationRecord
has_many :members
end
The Problem
Despite the simplicity of the models, the rules surrounding adding new members varies based on what app your are in, how the member is being added, and so on. In fact, there are 5 distinct ways in which a new member can be added:
- Via the admin interface (i.e., a typical form) in our app by an administrator
- Via the admin interface in our app by a client (this form is different than the administrator form)
- Via a spreadsheet with many (or just one) emails uploaded by the client
- Via a client’s public newsletter signup form
- Via a client’s newsletter, which has an option to “refer a friend”
Each of these ways of adding a new member has some slight variations in what they need to do to make sure a new member is being legitimately added. I don’t think it’s a big step for readers to imagine the sh$t-storm of conditional logic that would get thrown around five different controllers across three or four apps just to make sure all the right rules are followed, etc.
We had a lot of little issues over the years as our client and member base grew and we had more and more people adding members to their DB in all the different ways.
The Solution, Part I
I went through quite a few options and approaches to creating a consistent interface for adding new members to our system. The most important step in that process was stepping back and asking the following question: “what actions are common across all instances of adding a new member?”
What I found was that the following things were common and could be coded in the base Member model:
class Member < ApplicationRecord
include Verifiable
include EmailValidatable
validates :email, presence: true, email: true, if: proc { |object| object.new_record? || object.email_changed? }, uniqueness: { scope: :client_id, message: 'already subscribed' }
validates_with ExclusiveValidator, if: proc { |object| object.new_record? || object.email_changed? }
validates_with PermittedValidator, if: proc { |object| object.new_record? || object.email_changed? }
validates_with SpamTrapValidator, if: proc { |object| object.new_record? || object.email_changed? }
end
When a new member is added, we have to first verify it’s a legitimately formatted email address (the validates :email line). Next, we have to make sure that the email is actually a real, live, usable email and not some spam trap, etc. (this is performed via both the EmailValidtable module, which is really part of the validates :email method, and Verfiiable concern).
Once we know that the email is legit-legit, we can proceed to our next common steps and ensure it’s a permitted email (we do not allow role accounts, such as info@domainname.extension or sales@domainname.extension, for example), not an already found Spam Trap (in case FreshAddress missed it, we double-check our records) and, finally, we make sure that the email is not exclusive to another client in our system.
If all those rules are met, we can proceed with adding the new member to the system … assuming the specific rules for the 5 spots to add a new member, noted above, are also met. Note: I did not determine the business logic and, while some of it is just the result of the growth of a large, enterprise application over 10 years but some of it is just the result of business people trying to dictate coding rules … pretty sure most of us have been there LOL!
Plug: We use FreshAddress as our third-party email verification system and it is pretty dang impressive how many crap emails they can weed out for us. I’m just offering in case anyone else finds themselves in need of such a service and is not sure who is good and who isn’t.
The Solution, Part II
Now that I knew what commonality exists in the action of adding a new member, I had to figure out how in the eff I was going to pull it off. After a good bit of reading, searching, browsing, reading and then some more browsing and reading, I opted for a mountable Rails Engine.
After all that reading, I ended up being pretty uncreative and just used the Rails Guides - https://guides.rubyonrails.org/engines.html - to get me started. And it was plenty. I have to note that we in the Ruby and Ruby on Rails world are so fortunate to have such fantastically detailed documentation.
I chose a mountable engine because I needed to easily include it in several of our Rails apps and use it without having to refactor a ton of code. Basically, I needed as close to a plug and play solution as I could get.
The Engine, named the member_validator, has a base set of the models related to adding a new member to be used in each and every app in which it’s used. In total, the engine ended up with a set of 14 models, 3 concerns (2 for models and 1 for a controller in the including app) and a bevy of service objects.
One concern we had with mounting the engine was namespacing. Lots of engines will namespace their models in the including app with a format such as EngineName::ModelName.
Because this engine was being extracted from an existing, nearly-decade-old application (as well as going into three three-to-four-year-old microservices), we had to make sure that the models could be used in those apps without namespacing them (because there were 100s or 1000s of calls across all the apps that did not use any namespacing in the pre-engine world).
By mounting it and not using an engine-specific namespace, we were able to use the models in the existing apps without having to refactor a ton of code. Any call to Member.where(...) would still work in the existing apps and would not need to be changed to MemberValidator::Member.where(...).
Long and short, coding in the engine was really easy, especially since Rails gives you a “dummy” app in the Engine you can use to test, etc. But, in the end, I just “coded” the Engine like I would any Rails app (ok, just about any Rails app).
The API
The Engine exposes a five methods to the including app, which is all handled through a Controller Concern that is/will be included in the apps. The two that are used quite regularly are certify and pre_flight. The others are helpful but rarely used across the apps.
module Validatable
extend ActiveSupport::Concern
# To use these methods in the containing app:
# include Validatable
# returns an EmailAddress object
# verifies the email is legit across the board
def certify(member_params:)
member = member_params.is_a?(Member) ? member_params : Member.new(member_params)
Members::Certify.new(member_to_certify: member).call
end
# sometimes an email goes bad and we need to check it ... that's what this does
def recertify(member:, date_check: true)
Members::Recertify.new(member: member).call if date_check
end
# no validation requried (admin only feature!)
def manual_override(member:)
Members::ManualOverride.new(member: member).call
end
# just seeing if things are going to go well here
def spot_check(email)
Emails::VerificationResults.new(member: Member.new(email: email)).call
end
# this basically does a quickie check on the email address
def pre_flight(email, client_id)
Emails::PreFlight.new(email: email, client_id: client_id).call
end
end
The Solution, Part III
Now, we have the fun of integrating the Engine. Step one is pretty darn easy. Add your Engine, which is really just a gem, to your Gemfile:
gem 'member_validator'
Next, in a controller where you’d like to use the Engine, you would do the following:
class EmailProcessorsController < ApplicationController
include Validatable # this is the main thing!
helper EmailProcessorsHelper
def create
# here we are calling the pre_flight method from the Engine
@results = pre_flight(params[:email].strip.downcase, params[:client_id])
respond_to do |format|
format.turbo_stream {}
end
end
end
Here is the same in another controller but where we certify the email:
class ClientsController < ApplicationController
def create
# the certify method from the engine
@member = certify(member_params: member_params)
# do other stuff here :)
end
end
The Results
We’ve found that this new approach has reduced both the complexities in our codebase(s) as well as making the process of adding a new member far more consistent and reliable. It was pretty easy to build out and include the Engine in our apps once we figured out that the Engine was the way for us to go in terms of solving this particular problem.
One piece of general advice that I had to tell myself over and over: DO NOT ENGINE ALL THE THINGS! You, like me, may want to as you get started but, really, it’s the the solution for every problem like this. It just happened to be the right one for our app.
Gotchas/Issues
One thing we did run into with the move to an Engine was the models themselves. Using just our base Member model as an example, it had a different set of methods in it in one app versus another. Part of this was the model being used differently in each app but it was also a legacy issue from our early days of development when we followed the “Skinny Controller, Fat Model” approach.
Over time, this became, “ALL THE THINGS ARE SKINNY”. Now our models are merely wrappers for the table they represent (we do have a few, more robust, tableless models) and maybe a method or two to easily manipulate/present the data (e.g., the Member model has a full_name method that returns the first and last name of the member). However, for these older models, we could not just remove all the methods and logic because they were called from all over each app. We simply have not had enough time for such a refactoring.
Anyway, there was/is a nice solution: Overrides. In each app, we have a folder called app/overrides that includes just the models from the Engine that we need to override in the particular app:
# it feels backwards but the `instance_eval` is your class methods (i.e., self.method_name)
Member.instance_eval do
# we don't use all relationships in all apps so we can customize them here for this particular app's override of the model
has_many :other_model, dependent: :destroy
has_many :other_model_2, dependent: :destroy
# great example where we do not keep an audit log for all things in all apps and did not want to
# add PaperTrail to the Engine's gem dependencies since it's only used in a couple of the apps, we let the apps handle that.
has_paper_trail ignore: %i[created_at updated_at]
# this scope is only used in one app s...
scope :at_risk_members, lambda {
# fancy scope things here
}
end
# it feels backwards but the `class_eval` is your instance methods
Member.class_eval do
def phone_number
# a nifty way to get a phone number
end
end
To load your overrides, you only need to add the following code block to config/application.rb:
# overrides
overrides = Rails.root.join('app/overrides')
Rails.autoloaders.main.ignore(overrides)
config.to_prepare do
Dir.glob("#{overrides}/**/*_override.rb").each do |override|
load override
end
end
In some apps, having to add overrides could be a pain in the butt. Fortunately, it was pretty easy across our apps and, in most cases, we only override a couple models here and there from the Engine.
Screenshot of Engine File Structure
On its own, the Engine is really nothing more than an incredibly simple Rails app. I’ve included a screenshot of the file structure below just to illustrate how “normal” the Engine will look to a Rails developer. For me, seeing and understanding that it was, really, just another Rails app made it more approachable and less intimidating.
