{"success":true,"data":{"title":"Common Table Expressions in Ruby on Rails","slug":"common-table-expressions-in-ruby-on-rails","excerpt":"A practical look at Common Table Expressions in Ruby on Rails. Covers writing CTEs as raw SQL in Rails 7.0 and below, the N+1 query problem that raw SQL results introduce (because ActiveRecord's preload cannot be used), a workaround that calls the preloader directly, and the cleaner approach added in Rails 7.1 — the ActiveRecord-native \"with\" method, which generates the CTE and restores full preload support.","has_code_sample":true,"published_at":"2023-03-19T16:45:00.000Z","content":"\u003cp\u003eA Common Table Expression (CTE) is a temporary set of results created from a dataset in a database that gets used like any other table. They allow you to write complex queries by breaking them into more manageable (and reusable) blocks. For Rails versions \u0026lt;=7.0, we must use Common Table Expressions as raw SQL. Here’s an elementary example:\u003c/p\u003e\n\n\u003cdiv class=\"language-ruby highlighter-rouge\"\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre class=\"highlight\"\u003e\u003ccode\u003e\u003cspan class=\"n\"\u003esql\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026lt;\u0026lt;-\u003c/span\u003e\u003cspan class=\"no\"\u003eSQL\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003esquish\u003c/span\u003e\u003cspan class=\"sh\"\u003e\n  WITH not_in_team AS (\n    SELECT users.id FROM users\n    WHERE team_id IS NULL\n  )\n  SELECT * FROM users\n  WHERE id NOT IN(SELECT * FROM not_in_team)\n\u003c/span\u003e\u003cspan class=\"no\"\u003eSQL\u003c/span\u003e\n\n\u003cspan class=\"n\"\u003eresult\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"no\"\u003eUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003efind_by_sql\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003esql\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# User Load (4.6ms)  WITH not_in_team AS ( SELECT users.id FROM users WHERE team_id IS NULL ) SELECT * FROM users WHERE id NOT IN(SELECT * FROM not_in_team)\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/div\u003e\n\n\u003cp\u003eThe SQL isn’t important here…what is essential is that we have to construct a query using raw SQL. Writing a query in SQL is a problem for a couple of reasons:\u003c/p\u003e\n\n\u003cul\u003e\n  \u003cli\u003eIf we were using values provided by a user, we’d have to escape them manually.\u003c/li\u003e\n  \u003cli\u003eWe leave ourselves open to N+1 queries.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eLet’s look into the N+1 issue. If we wanted to iterate over the users and print out all of the names of the teams that they’re in, we might do something like this:\u003c/p\u003e\n\n\u003cdiv class=\"language-ruby highlighter-rouge\"\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre class=\"highlight\"\u003e\u003ccode\u003e\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003emap\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003e_1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003eteam\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ename\u003c/span\u003e \u003cspan class=\"p\"\u003e}.\u003c/span\u003e\u003cspan class=\"nf\"\u003ejoin\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e', '\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (1.1ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" = $1 LIMIT $2  [[\"id\", 1], [\"LIMIT\", 1]]\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (0.4ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" = $1 LIMIT $2  [[\"id\", 1], [\"LIMIT\", 1]]\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (0.3ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" = $1 LIMIT $2  [[\"id\", 2], [\"LIMIT\", 1]]\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (0.3ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" = $1 LIMIT $2  [[\"id\", 2], [\"LIMIT\", 1]]\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# =\u0026gt; \"A Team, A Team, B Team, B Team\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/div\u003e\n\n\u003cp\u003eThe code above would give the desired result, but for each user, it would perform an SQL query which could be more efficient. Usually, when writing an ActiveRecord query, we’d use \u003ccode\u003epreload\u003c/code\u003e, which would preload the \u003ccode\u003eteams\u003c/code\u003e association, thereby reducing the number of database calls from N+1 to 2. We can’t do that with raw SQL. There is a workaround — we can call the preloader directly:\u003c/p\u003e\n\n\u003cdiv class=\"language-ruby highlighter-rouge\"\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre class=\"highlight\"\u003e\u003ccode\u003e\u003cspan class=\"no\"\u003eActiveRecord\u003c/span\u003e\u003cspan class=\"o\"\u003e::\u003c/span\u003e\u003cspan class=\"no\"\u003eAssociations\u003c/span\u003e\u003cspan class=\"o\"\u003e::\u003c/span\u003e\u003cspan class=\"no\"\u003ePreloader\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"ss\"\u003erecords: \u003c/span\u003e\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"ss\"\u003eassociations: \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"ss\"\u003e:team\u003c/span\u003e\u003cspan class=\"p\"\u003e]).\u003c/span\u003e\u003cspan class=\"nf\"\u003ecall\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (0.8ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" IN ($1, $2)  [[\"id\", 1], [\"id\", 2]]\u003c/span\u003e\n\n\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003emap\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003e_1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003eteam\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ename\u003c/span\u003e \u003cspan class=\"p\"\u003e}.\u003c/span\u003e\u003cspan class=\"nf\"\u003ejoin\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e', '\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# =\u0026gt; \"A Team, A Team, B Team, B Team\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/div\u003e\n\n\u003cp\u003eManually calling the preloader solves the N+1 issue. It feels a little hacky, but it works. Luckily, Rails 7.1 will make this easier — in it, we can use ActiveRecord to define Common Table Expressions, which means we can also use \u003ccode\u003epreload\u003c/code\u003e.\u003c/p\u003e\n\n\u003cdiv class=\"language-ruby highlighter-rouge\"\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre class=\"highlight\"\u003e\u003ccode\u003e\u003cspan class=\"n\"\u003eresult\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"no\"\u003eUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"ss\"\u003enot_in_team: \u003c/span\u003e\u003cspan class=\"no\"\u003eUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ewhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"ss\"\u003eteam_id: \u003c/span\u003e\u003cspan class=\"kp\"\u003enil\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"nf\"\u003eselect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"ss\"\u003e:id\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n             \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ewhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e'users.id NOT IN(SELECT * FROM not_in_team)'\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n             \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003epreload\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"ss\"\u003e:team\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\n\u003cspan class=\"c1\"\u003e# User Load (0.7ms)  WITH \"not_in_team\" AS (SELECT \"users\".\"id\" FROM \"users\" WHERE \"users\".\"team_id\" IS NULL) SELECT \"users\".* FROM \"users\" WHERE (users.id NOT IN(SELECT * FROM not_in_team))\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# Team Load (0.5ms)  SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" IN ($1, $2)  [[\"id\", 1], [\"id\", 2]]\u003c/span\u003e\n\n\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003emap\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003e_1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003eteam\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nf\"\u003ename\u003c/span\u003e \u003cspan class=\"p\"\u003e}.\u003c/span\u003e\u003cspan class=\"nf\"\u003ejoin\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e', '\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003cspan class=\"c1\"\u003e# =\u0026gt; \"A Team, A Team, B Team, B Team\"\u003c/span\u003e\n\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/div\u003e\n\n\u003cp\u003eWe define the CTE using the \u003ccode\u003ewith\u003c/code\u003e method; we can then use it in the where condition as we did with the raw SQL. Notice that we can then use \u003ccode\u003epreload\u003c/code\u003e. Much cleaner, and no N+1!\u003c/p\u003e\n\n\u003cp\u003eThis is a really simplified example to give a bare bone demonstration of what’s coming. There is a \u003ca href=\"https://github.com/rails/rails/pull/37944\"\u003emuch more detailed discussion\u003c/a\u003e in the PR that added the functionality with more realistic examples if you’d like to dig a little deeper.\u003c/p\u003e\n","tags":["Databases","Ruby on Rails","SQL"]}}