If you are using Cloudflare, it can be helpful to configure Cloudflare to push request logs to S3. Otherwise, the Cloudflare dashboard provides only a limited view into your data (72 hours at a time and sampled data instead of full logs).

Once the Cloudflare request logs are in S3, they can be queried using Athena. This blog post even provides a nice CREATE TABLE command to set up the table in Athena.

However, there is a problem. When performing a query in Athena, it might have to scan all of the logs in S3, even if you try to limit the query. This can be slow and costly, as Athena queries are charged per byte scanned.

The only way to really limit the amount of data scanned is to partition the data.

This post assumes you have already set up Cloudflare to push logs to an S3 bucket, configured a database in Athena to access it, and then realized those logs will grow forever, along with your query times.

(If you just want the “how to” without the exposition, jump down to “Setting Up Partitions for Cloudflare Logs”.)

Partitioning

Most commonly, you will want to look at logs from a specific time period, so it makes sense to partition the logs by date.

Most of the partitioning documentation suggests the files (or in this case, S3 objects) include a column=value key pair in the name. The column can then be used as a partition.

Unfortunately, Cloudflare does not allow customizing the format of the file names it produces.

Fortunately, the file names do include date/time information. The logs are grouped by date and time range:

1

s3://mah_s3_bucket/20210812/20210812T223000Z_20210812T224000Z_9af500e2.log.gz

So all we need to do is grab that date “folder” name and that’s our partition! Easy, right?

No, wrong. This is AWS. Nothing is easy.

Use a Recurring Job?

Several of the AWS documentation pages suggest using ALTER TABLE to ADD PARTITIONs.

Something like this:

1
2

ALTER TABLE cloudflare_logs ADD
  PARTITION (dt = '2021-08-12') LOCATION 's3://mah_log_bucket/20210812/';

But since the logs will grow every day, we’ll need to add new a new partition every 24 hours. It is not possible to “predefine” the partitions.

This requires setting up a recurring job… somewhere… to periodically define the new partitions. So now we have to pull in another AWS service to make S3 and Athena work nicely?! No thanks!

Partition Projection

At the bottom of the page about partitions, there is a paragraph about ”partition projection” that sounds promising:

To avoid having to manage partitions, you can use partition projection. Partition projection is an option for highly partitioned tables whose structure is known in advance.

Yes, this is what we want! But how does it work?

Essentially like this:

We must define a column, its type, start and end values, and the interval between those values. Athena will then be able to extrapolate all the possible values.

Then we define a pattern to pull the value out of each S3 object name. This allows Athena to figure out which objects (logs) are associated with which partition value.

For example, the partition 20210812 will be associated with s3://mah_s3_bucket/20210812/20210812T223000Z_20210812T224000Z_9af500e2.log.gz

Once that’s all done, we can query based on the partition as if it were a column, like:

1
2
3

SELECT * FROM cloudflare_logs
WHERE log_date >= '20210812'
  AND log_date < '20210901';

Setting Up Partitions for Cloudflare Logs

Here are the steps that must be taken to set up the partitions:

1. Add the partition “column” when creating the table
2. Set several properties on the table to define the projection
3. Set the partition pattern to match against object names
4. Enable projection

(Actually, 2-4 are all the same: set (totally unvalidated) key-value properties on the table.)

Fortunately, all of this can be accomplished with one giant Athena command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

CREATE EXTERNAL TABLE `YOUR_TABLE_NAME`(
  `botscore` int,
  `botscoresrc` string,
  `cachecachestatus` string,
  `cacheresponsebytes` int,
  `cacheresponsestatus` int,
  `clientasn` int,
  `clientcountry` string,
  `clientdevicetype` string,
  `clientip` string,
  `clientipclass` string,
  `clientrequestbytes` int,
  `clientrequesthost` string,
  `clientrequestmethod` string,
  `clientrequestpath` string,
  `clientrequestprotocol` string,
  `clientrequestreferer` string,
  `clientrequesturi` string,
  `clientrequestuseragent` string,
  `clientsslcipher` string,
  `clientsslprotocol` string,
  `clientsrcport` int,
  `edgecolocode` string,
  `edgecoloid` int,
  `edgeendtimestamp` string,
  `edgepathingop` string,
  `edgepathingsrc` string,
  `edgepathingstatus` string,
  `edgeratelimitaction` string,
  `edgeratelimitid` int,
  `edgerequesthost` string,
  `edgeresponsebytes` int,
  `edgeresponsecontenttype` string,
  `edgeresponsestatus` int,
  `edgeserverip` string,
  `edgestarttimestamp` string,
  `firewallmatchesactions` array<string>,
  `firewallmatchesruleids` array<string>,
  `firewallmatchessources` array<string>,
  `originip` string,
  `originresponsestatus` int,
  `originresponsetime` int,
  `originsslprotocol` string,
  `rayid` string,
  `wafaction` string,
  `wafflags` string,
  `wafmatchedvar` string,
  `wafprofile` string,
  `wafruleid` string,
  `wafrulemessage` string,
  `workersubrequest` boolean,
  `zoneid` bigint)
PARTITIONED BY (
  `YOUR_COLUMN_NAME` string)
ROW FORMAT SERDE
  'org.openx.data.jsonserde.JsonSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION
  's3://YOUR_BUCKET_NAME/'
TBLPROPERTIES (
  'projection.enabled'='TRUE',
  'projection.YOUR_COLUMN_NAME.format'='yyyyMMdd',
  'projection.YOUR_COLUMN_NAME.interval'='1',
  'projection.YOUR_COLUMN_NAME.interval.unit'='DAYS',
  'projection.YOUR_COLUMN_NAME.range'='YOUR_START_DATE,NOW',
  'projection.YOUR_COLUMN_NAME.type'='date',

 'storage.location.template'='s3://YOUR_BUCKET_NAME/${YOUR_COLUMN_NAME}/'
)

These are the pieces related to partitioning:

1
2

PARTITIONED BY (
  `YOUR_COLUMN_NAME` string)

This tells Athena to set up a column for the partition.

(The type of the partition column must be string, even though the projection type must be date. Does this make any sense? NO.)

Then the table properties.

Turn on projection:

1

'projection.enabled'='TRUE'

Set the column type to date:

1

'projection.YOUR_COLUMN_NAME.type'='date'

This is so Athena knows how to interpolate values.

Define the date format to match:

1

'projection.YOUR_COLUMN_NAME.format'='yyyyMMdd'

Set the interval for the values to one day:

1
2

'projection.YOUR_COLUMN_NAME.interval'='1',
'projection.YOUR_COLUMN_NAME.interval.unit'='DAYS'

Set the range for the values:

1

'projection.YOUR_COLUMN_NAME.range'='YOUR_START_DATE,NOW'

NOW is a special value so the end of the range will always be the current day.

This sets a template to extract the date string from the object name, using the date template defined above, and setting the value in the column name specified for the projection:

1

'storage.location.template'='s3://YOUR_BUCKET_NAME/${YOUR_COLUMN_NAME}/'

If you are unfamiliar with Athena, it’s good to know that deleting/creating tables is low impact. If the table is already created, it is not a big deal to delete it and start over.

Here are the important bits above that you will need to change:

• YOUR_TABLE_NAME is whatever you want to name the table. Something like cloudflare_logs would probably make sense.
• YOUR_COLUMN_NAME is whatever you want to name the projection “column”. Could be dt like in the AWS docs, or log_date or whatever you want.
• YOUR_BUCKET_NAME is the name of the S3 bucket.
• YOUR START_DATE is the date of the first log. Something like 20210101.

The table should now indicate it is partitioned:

And the partition should show up as a column:

Using Date Partitions

To test if partitions are working as expected, a quick query like this will work:

1
2
3

SELECT DISTINCT(YOUR_COLUMN_NAME)
FROM "YOUR_TABLE_NAME"
LIMIT 10;

The expected output is several dates for which there are logs.

Once that is confirmed, the partition column can be used like any other column. Since the values is a basic ISO date format, comparison operators can be safely used even though the column is really just a string.

For a single day:

1
2

SELECT * FROM YOUR_TABLE
WHERE YOUR_COLUMN_NAME = "20200101";

For an inclusive range:

1
2
3

SELECT * FROM YOUR_TABLE
WHERE YOUR_COLUMN_NAME >= "20200101"
  AND YOUR_COLUMN_NAME <= "20210101";

And now your queries can be faster and cheaper!

The sound API in DragonRuby is a little tricky because at first it seems similar to the sprite API (e.g. using args.outputs) but it diverges sharply when you want to have some more control.

So let’s take a quick look at audio in DragonRuby!

Simple Music

As usual with DragonRuby, the simple stuff is simple.

To play a looping sound, for example as background music, get a sound file in .ogg format and then add it on to args.outputs.sounds:

1

args.outputs.sounds << 'sounds/my_music.ogg'

The audio will loop forever.

Simple Sound Effects

Want to fire off a single sound effect?

Get a sound file in .wav format and then add it on to args.outputs.sounds:

1

args.outputs.sounds << 'sounds/my_effect.wav'

The sound will play once.

Wait so the API behaves differently based on the file format?

Yep. A little odd but it works.

Adjusting the Volume Knob

Playing a sound effect in DragonRuby is pretty easy. But what if we need to adjust the volume or some other attribute of the audio? Or maybe loop a .wav or not loop a .ogg file?

You might think, based on how sprites work in DragonRuby, you could do something like this:

1
2
3
4
5
6

# This does not work!
args.outputs.sounds << {
  input: 'sounds/my_music.ogg',
  gain: 0.5,      # Half volume
  looping: false  # Don't loop
}

But this does not work.

Instead, you need to use an entirely different interface: args.audio.

args.audio is a hash table of sounds.

To add a new sound, assign it with a name:

1
2
3
4
5

args.audio[:music] = {
  input: 'sounds/my_music.ogg',
  gain: 0.5,
  looping: false
}

To adjust an attribute of the audio, access it by the same name:

1
2

# Turn up the volume
args.audio[:music].gain = 0.75

To pause a sound:

1

args.audio[:music].paused = true

To completely remove a sound:

1

args.audio.delete :music

More Music Manipulation

This is the full set of options for audio from the documentation:

1
2
3
4
5
6
7
8

args.audio[:my_music] = {
  input: 'sounds/my_music.ogg',
  x: 0.0, y: 0.0, z: 0.0,   # Relative position to the listener, x, y, z from -1.0 to 1.0
  gain: 1.0,                # Volume (0.0 to 1.0)
  pitch: 1.0,               # Pitch of the sound (1.0 = original pitch)
  paused: false,            # Set to true to pause
  looping: false,           # Set to true to loop
}

That’s it!

Next up in my notes on DragonRuby: render targets!

Weirdly, the documentation on DragonRuby’s render targets is limited to example code. Personally, I prefer prose when I am trying to learn… so here we are!

In DragonRuby, a render target is like an infinite canvas you can render as many regular sprites onto as you want, then manipulate the whole thing as if it is one sprite.

This is especially good for things like tiled backgrounds that are built once and do not change.

Let’s take an example.

Clouds!

Let’s start off very simple and build up.

First, here’s all the code to render a single 250x250 pixel image to the screen:

1
2
3
4
5
6
7
8
9
10
11

def tick args
  clouds = {
    x: 0,
    y: 0,
    h: 250,
    w: 250,
    path: 'sprites/cloud.png'
  }

  args.outputs.sprites << clouds
end

More Clouds

Cool, but I’d like to fill the whole window with clouds, so I’m going to tile them.

The code below makes a 6x3 grid of the cloud image.

(In DragonRuby, the screen is always 1280x720. Our grid is 1500x750 but I’m not trying to be too precise with the numbers here.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def tick args
  clouds = []

  6.times do |x|
    3.times do |y|
      clouds << {
        x: x * 250,
        y: y * 250,
        h: 250,
        w: 250,
        path: 'sprites/cloud.png'
      }
    end
  end

  args.outputs.sprites << clouds
end

Ah… blue clouds. Nice.

On every tick, the code builds up an array of 18 sprites (images) and renders it out to the screen.

(There are a number of ways to make this more efficient - check out the previous posts in this series for different ways to “cache” the sprite information.)

Render Targets

But in this post we are talking about render targets - which is a way of rendering a bunch of sprites (or any other renderable thing) just once, and then treating the whole group of sprites as a single sprite. This is faster, simpler, and enables some neat effects.

The code only needs minor changes to switch the cloud grid to using a render target instead:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# Move cloud grid creation into a helper method
def make_clouds(args)
  clouds = []

  6.times do |x|
    3.times do |y|
      clouds << {
        x: x * 250,
        y: y * 250,
        h: 250,
        w: 250,
        path: 'sprites/cloud.png'
      }
    end
  end

  # Similar to `args.outputs`,
  # render targets have `.sprites`, `.solids`, etc.
  # The argument will be used as the path below
  args.render_target(:clouds).sprites << clouds
end

def tick(args)
  # Set up the render target on the first tick
  if args.tick_count == 0
    make_clouds(args)
  end

  # Output a single sprite
  # located at 0,0 and the size of the whole grid
  # created in `make_clouds`
  args.outputs.sprites << {
    x: 0,
    y: 0,
    w: 250 * 6,
    h: 250 * 3,
    path: :clouds # Name of the render target!
  }
end

For convenience, the code above moves the creation of the cloud grid and the render target into a helper method which gets called on the first tick of the game.

args.render_target(:clouds) automatically creates a new render target named :clouds if it does not already exist. Then we can render things to it just as if it were args.outputs.

Interestingly, render targets do not seem to have an innate width or height. In order to avoid unintentional scaling, you will need to “know” how big the render target is. In this case, we know it is a 6x3 grid of 250x250 images, so the size is fairly straightforward. I left the math in to make it clearer.

Finally, we reference the render target similarly to an image file, but pass in the name of the render target as the :path instead of an actual file path.

Static Sprites, Too!

As explored in a different post, we can use static_sprites to “render” the sprite once.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# No changes here
def make_clouds(args)
  clouds = []

  6.times do |x|
    3.times do |y|
      clouds << {
        x: x * 250,
        y: y * 250,
        h: 250,
        w: 250,
        path: 'sprites/cloud.png'
      }
    end
  end

  args.render_target(:clouds).sprites << clouds
end

def tick(args)
  if args.tick_count == 0
    make_clouds(args)

    # Create the clouds sprite once
    # and keep it in `args.state`.
    args.state.clouds = {
      x: 0,
      y: 0,
      w: 250 * 6,
      h: 250 * 3,
      path: :clouds
    }

    # Add the clouds sprite just once
    # as a "static" sprite
    args.outputs.static_sprites << args.state.clouds
  end
end

And now we can move the clouds around just by changing the attributes on the render target.

Adding a little bit of code at the end of tick:

1

  args.state.clouds.x = (Math.sin(args.tick_count / 20) * 100) - 100

(The calculation and numbers aren’t really important here, I just fiddled around until something looked decent.)

Oh hey! Those extra pixels on the sides of the cloud grid actually came in handy.

What Else?

Remember, the entire render target is like one sprite now. That means all the regular sprite attributes (e.g. color, size, blending, flipping, rotation) can be applied to the entire thing at once.

Wait, did you say rotation?

Sure, let’s make ourselves dizzy.

1

  args.state.clouds.angle = (Math.sin(args.tick_count / 120) * 180)

Okay, that’s as deep as we’ll go on render targets in this post!

I enjoy playing with the DragonRuby Game Toolkit, but the documentation and many of the examples are very much intended for non-Rubyists. Additionally, as a game engine, it’s more data/functionally-oriented than most Rubyists are used to. For example, the main game loop in the tick method needs to be implemented as a top-level method.

This post walks through structuring a game in a way that is a little more familiar to Rubyists.

Caveat! I am new to DragonRuby myself and this is not meant to be the “correct” or “best” or even “great” way to organize your code. It’s just a pattern I’ve started using and it might be useful for you!

(By the way, while DragonRuby is a commercial product, you can often grab a free copy. Keep an eye out for sales!)

Starting Example

First, let’s start with some code that isn’t using any class definitions at all. Everything happens inside tick:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def tick(args)
  # Set up player object
  args.state.player ||= {
    x: args.grid.w / 2,
    y: args.grid.h / 2,
    w: 20,
    h: 20,
  }

  # Move player based on keyboard input
  if args.inputs.keyboard.left
    args.state.player.x -= 10
  elsif args.inputs.keyboard.right
    args.state.player.x += 10
  elsif args.inputs.keyboard.down
    args.state.player.y -= 10
  elsif args.inputs.keyboard.up
    args.state.player.y += 10
  end

  # Render the "player" as a square to the screen
  args.outputs.solids << args.state.player
end

As a reminder, DragonRuby automatically loads up code from a file called mygame/app/main.rb and then calls tick 60 times per second.

args.state is like an OpenStruct where you can add on whatever attributes you want and store whatever you would like. In this case, we add a hash that we name player.

The code then checks for keyboard input and adjusts the position of the “player”. (To keep things very simple, we don’t worry about keeping the player on the screen.)

Finally, we render the “player” as a solid square.

This is simple enough and the code isn’t too complicated. But, just for fun, let’s slowly transform it to be a little more “object-oriented”.

Game Object

First, let’s create a main “game” object to hold our logic, instead of putting it all in tick.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

class MyGame
  # Adds convenience methods for args, gtk, keyboard, etc.
  attr_gtk

  def initialize(args)
    args.state.player = {
      x: args.grid.w / 2,
      y: args.grid.h / 2,
      w: 20,
      h: 20,
    }
  end

  def tick
    if keyboard.left
      state.player.x -= 10
    elsif keyboard.right
      state.player.x += 10
    elsif keyboard.down
      state.player.y -= 10
    elsif keyboard.up
      state.player.y += 10
    end

    outputs.solids << state.player
  end
end

def tick args
  $my_game ||= MyGame.new(args)
  $my_game.args = args
  $my_game.tick
end

Now the tick method only sets up the global $my_game on the first tick, then sets args on each tick and calls the game’s tick method.

(Tangent alert! Is it necessary to set args on every tick? Not strictly - you could set self.args = args in initialize and it will work okay. But if you want to use DragonRuby’s unit test framework, it may cause problems because each test has a fresh copy of args.)

Using attr_gtk allows the code to be a bit shorter. args, state, keyboard, and more now have convenience methods for them.

Instance Variables Instead of State

args.state is essentially a global variable space. This is a big convenience when a game is all top-level methods - otherwise you would have to figure out where to stash all your game state yourself.

However, it’s not required to use it.

The code below uses @player to store the player hash, instead of args.state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

class MyGame
  attr_gtk
  attr_reader :player

  def initialize(args)
    @player = {
      x: args.grid.w / 2,
      y: args.grid.h / 2,
      w: 20,
      h: 20,
    }
  end

  def tick
    if keyboard.left
      player.x -= 10
    elsif keyboard.right
      player.x += 10
    elsif keyboard.down
      player.y -= 10
    elsif keyboard.up
      player.y += 10
    end

    outputs.solids << player
  end
end

def tick args
  $my_game ||= MyGame.new(args)
  $my_game.args = args
  $my_game.tick
end

One thing that has thrown me off with DragonRuby is understanding just how much “regular” Ruby I can use. For the most part, other than how the tick method is used as the main game loop, you can use the Ruby language constructs you are comfortable with.

Splitting Things Up

No big change here, but as a game grows it’s easier to split the steps of each game loop into different methods.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

class MyGame
  attr_gtk
  attr_reader :player

  def initialize(args)
    @player = {
      x: args.grid.w / 2,
      y: args.grid.h / 2,
      w: 20,
      h: 20,
    }
  end

  def tick
    handle_input
    render
  end

  def handle_input
    if keyboard.left
      player.x -= 10
    elsif keyboard.right
      player.x += 10
    elsif keyboard.down
      player.y -= 10
    elsif keyboard.up
      player.y += 10
    end
  end

  def render
    outputs.solids << player
  end
end

def tick args
  $my_game ||= MyGame.new(args)
  $my_game.args = args
  $my_game.tick
end

Player Class

Final step in this post - let’s move the “player” out to a separate class.

In this example, it might not make a lot of sense. But in most games there will be a lot of state and logic you might want to associate with the “player” or any other objects in the game. Having it be its own class helps keep the logic in one place.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

class MyGame
  attr_gtk
  attr_reader :player

  def initialize(args)
    @player = Player.new(args.grid.w / 2, args.grid.h / 2)
  end

  def tick
    handle_input
    render
  end

  def handle_input
    if keyboard.left
      player.x -= 10
    elsif keyboard.right
      player.x += 10
    elsif keyboard.down
      player.y -= 10
    elsif keyboard.up
      player.y += 10
    end
  end

  def render
    outputs.solids << player
  end
end

class Player
  attr_sprite

  def initialize(x, y)
    @x = x
    @y = y
    @w = 20
    @h = 20
  end
end

def tick args
  $my_game ||= MyGame.new(args)
  $my_game.args = args
  $my_game.tick
end

Here the code uses another DragonRuby convenience. attr_sprite adds a bunch of helper methods that allow you to use any object as a sprite/solid/border, etc. (Note that the code still passes player into outputs.solids and DragonRuby treats it as a solid. If it were passed into outputs.sprites then it would be treated like a sprite instead!)

Separate Files?

For the sake of a blog post, all the code is together. But there is no reason not to start splitting the code across separate files.

But! In DragonRuby there is one weirdness with require: you must include the file extension (usually .rb), while in regular Ruby that is usually omitted.

Wrapping Up

Did our code become longer and less straight-forward? Yes, for this small example, definitely.

But as a game (or any project) grows, pulling bits out into modular pieces is going to be an advantage. Personally I fall back on this structure pretty quickly when I start a new DragonRuby project.

Hopefully this post is useful to Rubyists trying to get into DragonRuby!

In a previous post we looked at different ways to render outputs (sprites, rectangles, lines, etc.) in the DragonRuby Game Toolkit.

The post ended by hinting at a more efficient way to render outputs instead of adding them to e.g. args.outputs.solids or args.outputs.sprites each tick.

This post explores the world of “static outputs”!

Static What?

First of all, we should address the most confusing part of all this.

“Static” does not mean the images don’t move or change. Instead, it means that render “queue” is not cleared after every tick.

Normally, one would load up the queue each tick, like this:

1
2
3
4
5
6
7
8
9
10

def tick args

  # Render a black rectangle
  args.outputs.solids << {
    x: 100,
    y: 200,
    w: 300,  # width
    h: 400   # height
  }
end

But this is kind of wasteful. We are creating a new hash table each tick (60 ticks/second) and then throwing it away. Also each tick we are filling up the args.outputs.solids queue and then emptying it.

Instead, why not create the hash table once, load up the queue once, and then re-use them?

That’s the idea of static outputs!

There are static versions for each rendered type:

• args.outputs.static_borders
• args.outputs.static_labels
• args.outputs.static_primitives
• args.outputs.static_solids
• args.outputs.static_sprites

Going Static

Starting Out

Here’s an example with comments explaining what the code is doing. This “game” simply moves a square back and forth across the screen. This is the entire program!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

def tick args
  # Initialize the x location of the square
  args.state.x ||= 0

  # Initialize the direction/velocity
  args.state.direction ||= 10

  # If we hit the sides, change direction
  if args.state.x > args.grid.right or args.state.x < args.grid.left
    args.state.direction = -args.state.direction
  end

  # Update the x location
  args.state.x += args.state.direction

  # Build the square
  square = {
    x: args.state.x,
    y: 400,
    w: 40,
    h: 40,
  }

  # Add the square to the render queue
  args.outputs.solids << square
end

The resulting output looks like:

This example introduces args.state. This is basically a persistent bag you can throw anything into. (For Rubyists - this is like OpenStruct.)

x and direction are not special, they are just variables we are defining. We use ||= to initialize them because we only want to set the values on the first tick.

This example illustrates the point from above - every tick it creates a new square and adds it to the queue. The queue is emptied out and then the code starts all over again.

Seems wasteful, right?

Caching the Objects

First thing I think of is - “why not create the square once, then just update the object each tick? Does that work?” Yes! It does.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def tick args
  args.state.direction ||= 10
  args.state.square ||= {
    x: 0,
    y: 400,
    w: 40,
    h: 40,
  }

  if args.state.square[:x] > args.grid.right or args.state.square[:x] < args.grid.left
    args.state.direction = -args.state.direction
  end

  args.state.square[:x] += args.state.direction

  args.outputs.solids << args.state.square
end

In this code, we create the square only once and then store it in args.state.square.

Instead of having a separate x variable, the code updates the x property on the square directly.

This is better, but we are still updating args.outputs.solids each tick.

Full Static

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

def tick args
  args.state.direction ||= 10
  args.state.square ||= {
    x: 0,
    y: 400,
    w: 40,
    h: 40,
  }

  # On the first tick, add the square to the render queue
  if args.tick_count == 0
    args.outputs.static_solids << args.state.square
  end

  if args.state.square[:x] > args.grid.right or args.state.square[:x] < args.grid.left
    args.state.direction = -args.state.direction
  end

  args.state.square[:x] += args.state.direction
end

In this code, we use the fact that the first args.tick_count is 0 to add the square to args.outputs.static_solids just once. It will continue to be rendered on each tick.

Performance

Intuitively, since the code is doing less, it should be faster. But does it really make a difference?

It depends on your game, how much it’s doing per tick, how many sprites you are rendering, and what platform/hardware it’s running on.

The examples above? Not going to see any difference using static_solids.

But DragonRuby contains two examples that directly compare args.outputs.sprites vs. args.outputs.static_sprites (here and here).

In these examples, you can play with the number of “stars” rendered to see different performance. On my ancient laptop, I do not see a performance difference until around 3,000 stars.

Your mileage may vary, though!

Should I Always Use the Static Versions?

It depends! Probably not?

If your code mainly manipulates the same objects around the screen and always renders them in the same order, then using the static_ approach might be simpler and faster.

But in many cases it might be easier to simply set up the render queues each tick, especially if the objects rendered or their ordering change regularly. Otherwise, managing the state of the rendering queues can become cumbersome. (We haven’t even talked about clearing the static queues, for example.)

Some of this comes down to personal preference and how you would like to structure your code. But hopefully this post has helped explain how to use the args.outputs.static_* methods in your game!

DragonRuby Game Toolkit (DRGTK) is a 2D game engine built with mRuby, SDL, and LLVM. It’s meant to be tiny, fast, and allow you to turn out games quickly using Ruby.

Unfortunately, since the documentation is focused on making games quickly, I sometimes get lost when trying to figure out how to do things that should be simple. DragonRuby seems to have borrowed Perl’s ”There’s more than one way to do it” philosophy because for anything you want to do with the API there are several ways to do it.

The documentation and examples tend to focus on the simplest forms (which is fine) but then require digging and experimentation to figure out the rest.

To help explain/document the different API options, this post will go through different methods of rendering images (well, rectangles mostly).

API Levels

Level 0 - Getting Started

The main object one interacts with in DragonRuby is canonically called args (always accessible with $gtk.args… because there’s more than one way!)

To output things, like shapes, sprites, or sounds, you can use the “shovel” operator << on args.outputs - like args.outputs.sprites or args.outputs.sounds.

For these “basic” objects, you’ll need to shovel things in on every “tick” of the game engine.

This is a complete DragonRuby example to output a rectangle:

1
2
3

def tick args
  args.outputs.solids << [100, 200, 300, 400]
end

So far, so good.

(You can assume the rest of the examples below are inside a tick method if it’s not explicitly defined.)

Level 1 - Arrays

The documentation usually starts off by passing things to args.outputs as arrays - essentially positional arguments.

For example:

1

args.outputs.solids << [100, 200, 300, 400]

What does that do? I’m not quite sure!

Okay - it shows a black rectangle on the screen. Not that exciting, but useful enough for our examples.

The problem with using arrays though is remembering which index in the array is which attribute. On top of that, it’s not even recommended to pass in arrays because they are slow (for some reason).

Level 2 - Hashes

What is better than arrays? Hashes! (Hash tables/associative arrays for anyone not familiar with Ruby.)

1
2
3
4
5
6

args.outputs.solids << {
  x: 100,
  y: 200,
  w: 300,  # width
  h: 400   # height
}

Okay, that’s way easier to understand!

And there are more options, too:

1
2
3
4
5
6
7
8
9
10
11

args.outputs.solids << {
  x: 100,
  y: 200,
  w: 300,
  h: 400,
  r: 255,  # red
  g: 200,  # green
  b: 255,  # blue
  a: 100,  # alpha
  blendmode_enum: 0  # blend mode
}

Level 3 - Primitives

But there is yet another way… instead of using args.outputs.solids, args.outputs.labels, args.outputs.sprites, etc., we can output a hash to args.outputs.primitives but mark it as the right primitive “type”:

1
2
3
4
5
6
7
8
9
10
11

args.outputs.primitives << {
  x: 100,
  y: 200,
  w: 300,
  h: 400,
  r: 255,  # red
  g: 200,  # green
  b: 255,  # blue
  a: 100,  # alpha
  blendmode_enum: 0  # blend mode
}.solid!

Weird, but okay. Why might one want to do this? See the “Layers” section down below!

Level 4 - Classes

Finally, probably the most natural for a Rubyist: just use a class!

To do this, you must define all the methods expected for the type of primitive, plus define a method called primitive_marker that returns the type of primitive.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class ACoolSolid
  attr_reader :x, :y, :w, :h, :r, :g, :b, :a, :blendmode_enum

  def initialize x, y, w, h
    @x = x
    @y = y
    @w = w
    @h = h
  end

  def primitive_marker
    :solid
  end
end

def tick args
  args.outputs.primitives << ACoolSolid.new(100, 200, 300, 400)
end

Instead of defining a bunch of methods with attr_reader, you can use attr_sprite instead which is a DragonRuby shortcut method to do the same thing.

Layers

DragonRuby renders outputs in this order (from back to front):

• Solids
• Sprites
• Primitives
• Labels
• Lines
• Borders

For each “layer” the objects are rendered in FIFO order - the first things in the queue are rendered first.

But wait… one of these things is not like the others. Doesn’t primitives just hold things like solids, sprites, labels…?

Yes!

But using primitives enables better control over render order.

For example, what if we want to render a rectangle on top of a sprite? With the fixed rendering order above, it’s impossible! But by using args.outputs.primitives we can do it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def tick args
  a_solid = {
    x: 100,
    y: 200,
    w: 300,
    h: 400
  }.solid!

  a_sprite = {
    x: 100,
    y: 200,
    w: 500,
    h: 500,
    path: 'metadata/icon.png'
  }.sprite!

  args.outputs.primitives << a_sprite << a_solid
end

And here’s the proof:

Every Tick?

args.outputs.sprites, etc. get cleared after each call to tick. So every tick we have to recreate all the objects and pass them in to args.outputs. Seems wasteful, right? Yes, it is!

It’s somewhat odd that most DragonRuby examples show creating arrays or hashes for primitives each tick. It made me think somehow the rendering process was destructive - were the things added into args.outputs destroyed or modified in some way?

Turns out, no. It is fine to create e.g. a sprite representation once and render the same object each time.

Here we’ll use a global for demonstration purposes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

class ACoolSolid
  attr_reader :x, :y, :w, :h, :r, :g, :b, :a, :blendmode_enum

  def initialize x, y, w, h
    @x = x
    @y = y
    @w = w
    @h = h
  end

  def primitive_marker
    :solid
  end
end

$a_solid = ACoolSolid.new(10, 20, 30, 40)

def tick args
  args.outputs.primitives << $a_solid
end

But Wait…

We are still shoveling an object into args.outputs.primitives each time. Surely that is unnecessary?

Correct! There are static versions for each args.outputs (e.g. args.outputs.static_solids) that do not get cleared every tick.

Naturally, this is more efficient than creating objects and updating the outputs 60 times per second.

We’ll explore these options in a future post, but be aware they are available!

A while ago, I came across a Brakeman false positive that I wanted to fix.

For just one false positive, it became a bit of an epic journey.

The code looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

class Task < ApplicationRecord
  enum status: {
    pending: 0,
    success: 1,
    failed: 3,
  }
  NOT_FAILURES = ['pending', 'success'].freeze
end

class TaskRunner
  def get_failures
    start_time = Date.beginning_of_quarter
    end_time = Date.end_of_quarter

    no_failure_enums = Task.statuses.values_at(*Task::NOT_FAILURES)

    query = <<~QUERY
      SELECT COUNT(*)
      FROM `tasks`
      WHERE `tasks`.`status` NOT IN (#{no_failure_enums.join(',')})
      AND `tasks`.`time_end` BETWEEN #{start_time} AND #{end_time}
    QUERY

    # Line below triggers a SQL injection warning
    Task.connection.select_all(query)
  end
end

(You can imagine in reality the query is a bit more complicated and justifies writing it this way.)

This code results in an SQL injection warning from Brakeman:

1
2
3
4
5
6
7

Confidence: High
Category: SQL Injection
Check: SQL
Message: Possible SQL injection
Code: Task.connection.select_all("SELECT COUNT(*)\nFROM `filings`\nWHERE `filings`.`status` NOT IN (#{Task.statuses.values_at(*["pending", "success"]).join(",")})\nAND `filings`.`time_end` BETWEEN #{Date.beginning_of_quarter} AND #{Date.end_of_quarter}\n")
File: app/models/task.rb
Line: 25

This is a little hard to read, so let’s take a look at a better formatted version of the code Brakeman is complaining about:

1
2
3
4
5
6
7

Task.connection.select_all(
  "SELECT COUNT(*) \
  FROM `filings` \
  WHERE `filings`.`status` NOT IN (#{Task.statuses.values_at(*["pending", "success"]).join(",")}) \
  AND `filings`.`time_end` BETWEEN #{Date.beginning_of_quarter} \
  AND #{Date.end_of_quarter}"
)

Brakeman is warning about this SQL query because it is using string interpolation to unsafely add in values to the query. If an attacker could control those values, they could modify the SQL run by the database.

In particular, it’s warning about

1

Task.statuses.values_at(*["pending", "success"]).join(",")

(the value in no_failure_enums.join(',')).

However, in this case, we know that Task.statuses is actually a constant - it’s defined using enum in the Task class. This code is just grabbing the integer values for the given enums and joining them back into a comma-separated string.

So how do we get Brakeman to understand that this value is actually safe?

Splatted Arrays

Let’s dive in!

The call we care about:

1

Task.statuses.values_at(*["pending", "success"]).join(",")

First up is *["pending", "success"]. This code converts an array of strings to individual method arguments (i.e., values_at("pending", "success").

This is pretty easy to handle. In the case where a splatted array is the only argument to a method, just use the elements of the array as the argument list. (Check out the pull request here)

This gets us to:

1

Task.statuses.values_at("pending", "success").join(",")

Better!

Hash Values

In this case, values_at is Hash#values_at - it returns an array of values from the hash table for the given keys.

This is also not too difficult to implement. I went ahead and covered Hash#values at the same time. (Check out the pull request here)

Brakeman will now do something like this:

1
2

h = { a: 1, b: 2, c: 3 }
h.values_at(:a, :c)  #=> [1, 3]

Great! Back to our code sample, how does it look now?

1

Task.statuses.values_at("pending", "success").join(",")

Oh… it looks exactly the same because Brakeman has no idea what Task.statuses is.

Okay, no problem. We just need to implement support for ActiveRecord’s enum.

Detour!

Here I took a little detour. Task.statuses is a method that returns a hash value. Instead of just implementing that, I thought this would be a good time to support methods with single, simple return values.

For example:

1
2
3
4
5

class Dog
  def self.sound
    'bark'
  end
end

If Brakeman could know that Dog.sound returns 'bark', I could implement enums as method definitions that return simple arrays or hashes. (More on this later!)

To implement this functionality, I rewrote how Brakeman tracks methods (as real objects) and updated some method lookup code.

The details aren’t particularly interesting, but the code changes are here.

Back to Enums

Calling enum essentially defines a bunch of methods.

For example this code:

1
2
3
4
5
6
7

class Task < ApplicationRecord
  enum status: {
    pending: 0,
    success: 1,
    failed: 3,
  }
end

Will define methods like

1
2
3
4
5
6

Task.statuses # the one we care about!
Task.status
Task#status
Task#pending?
Task#success?
# ..etc.

You can pass in an explicit hash mapping keys to values or just an array of keys and Rails will do the mapping.

To implement this in Brakeman, we simulate the creation of the status and statuses methods:

1
2
3
4
5
6
7
8
9

class Task < ApplicationRecord
  def self.statuses
    {
      pending: 0,
      success: 1,
      failed: 3,
    }
  end
end

Then rely on the previous changes for Task.statuses now returning a hash.

Another Detour

You might have noticed that the enum definition uses status but we need statuses.

This required a tiny tweak to Brakeman’s extremely over-simplified pluralize.

Back to the False Positive

Where are we now?

With enum support and proper pluralization, this code:

1

Task.statuses.values_at(*["pending", "success"]).join(",")

Gets reduced like this:

1
2
3
4
5

{
  pending: 0,
  success: 1,
  failed: 3,
}.values_at(*["pending", "success"]).join(",")

to

1
2
3
4
5

{
  pending: 0,
  success: 1,
  failed: 3,
}.values_at("pending", "success").join(",")

to

1

[:BRAKEMAN_SAFE_LITERAL, :BRAKEMAN_SAFE_LITERAL].join(",")

to

1

"BRAKEMAN_SAFE_LITERAL,BRAKEMAN_SAFE_LITERAL"

Well… it’s not perfect. Note that the array values are strings, but our enum uses symbol keys. But Brakeman knows the enum is all literal values which are safe, so we end up with this.

The query now looks like:

1
2
3
4
5
6
7

Task.connection.select_all(
  "SELECT COUNT(*) \
  FROM `filings` \
  WHERE `filings`.`status` NOT IN (#{"BRAKEMAN_SAFE_LITERAL,BRAKEMAN_SAFE_LITERAL"}) \
  AND `filings`.`time_end` BETWEEN #{Date.beginning_of_quarter} \
  AND #{Date.end_of_quarter}"
)

Again, not perfect but at least Brakeman isn’t going to warn about interpolating a string literal into the query.

Not Done Yet

Ah, but wait. The warning is not gone!

1
2
3
4
5
6
7

Confidence: Medium
Category: SQL Injection
Check: SQL
Message: Possible SQL injection
Code: Task.connection.select_all("SELECT COUNT(*)\nFROM `filings`\nWHERE `filings`.`status` NOT IN (#{"BRAKEMAN_SAFE_LITERAL,BRAKEMAN_SAFE_LITERAL"})\nAND `filings`.`time_end` BETWEEN #{Date.beginning_of_quarter} AND #{Date.end_of_quarter}\n")
File: app/models/task.rb
Line: 25

What’s wrong now?

Brakeman doesn’t know what Date.beginning_of_quarter or Date.end_of_quarter are, so it generates a lower confidence warning about it. For SQL injection, Brakeman is pretty paranoid about any string interpolation, even if it’s not sure the values are “dangerous”.

But anything coming from Date is likely to be safe, so now Brakeman ignores Date calls in SQL.

Whew. Done?

Yep - now that code will no longer warn.

Except… in the months it took me to address this false positive, the code has changed in such a way that Brakeman now has a false negative problem (should be warning about some things, but isn’t). But that’s a different problem for a different day.

At least that one false positive is fixed!

Since early 2013, I have been maintaining rails-sqli.org, a collection of Rails ActiveRecord methods that can be vulnerable to SQL injection.

Rails 6 has been out since December 2019, but sadly the site has been missing information about changes and new methods in Rails 6.

As that deficiency has recently been rectified, let’s walk through what has changed since Rails 5!

delete_all, destroy_all

In earlier versions of Rails, delete_all and destroy_all could be passed a string of raw SQL.

In Rails 6, these two methods no longer accept any arguments.

Instead, you can use…

delete_by, destroy_by

New in Rails 6, delete_by and destroy_by accept the same type of arguments as where: a Hash, an Array, or a raw SQL String.

This means they are vulnerable to the same kind of SQL injection.

For example:

1
2

params[:id] = "1) OR 1=1--"
User.delete_by("id = #{params[:id]}")

Resulting query that deletes all users:

1

DELETE FROM "users" WHERE (id = 1) OR 1=1--)

order, reorder

Prior to Rails 6, it was possible to pass arbitrary SQL to the order and reorder methods.

Since Rails did not offer an easy way of setting sort direction, this kind of code was common:

1

User.order("name #{params[:direction]}")

In Rails 6.0, injection attempts would raise a deprecation warning:

1

DEPRECATION WARNING: Dangerous query method (method whose arguments are used as raw SQL) called with non-attribute argument(s): "--". Non-attribute arguments will be disallowed in Rails 6.1. This method should not be called with user-provided values, such as request parameters or model attributes. Known-safe values can be passed by wrapping them in Arel.sql().

Starting with Rails 6.1, some logic to check the arguments to order. If the arguments do not appear to be column names or sort order, they will be rejected:

1
2
3
4

> User.order("name ARGLBARGHL")
Traceback (most recent call last):
        1: from (irb):12
ActiveRecord::UnknownAttributeReference (Query method called with non-attribute argument(s): "name ARGLBARGHL")

It is still possible to inject additional columns to extract some information from the table, such as number of columns or names of the columns:

1
2

params[:direction] = ", 8"
User.order("name #{params[:direction]}")

Resulting exception:

1

ActiveRecord::StatementInvalid (SQLite3::SQLException: 2nd ORDER BY term out of range - should be between 1 and 7)

pluck

pluck pulls out specified columns from a query, instead of loading whole records.

In previous versions of Rails, pluck (somewhat surprisingly!) accepted arbitrary SQL strings if they were passed in as an array.

Like order/reorder, Rails 6.0 started warning about this:

1
2

 > User.pluck(["1"])
DEPRECATION WARNING: Dangerous query method (method whose arguments are used as raw SQL) called with non-attribute argument(s): ["1"]. Non-attribute arguments will be disallowed in Rails 6.1. This method should not be called with user-provided values, such as request parameters or model attributes. Known-safe values can be passed by wrapping them in Arel.sql().

In Rails 6.1, pluck now only accepts attribute names!

reselect

Rails 6 introduced reselect, which allows one to completely replace the SELECT clause of a query. Like select, it accepts any SQL string. Since SELECT is at the very beginning of the SQL query, it makes it a great target for SQL injection.

1
2

params[:column] = "* FROM orders -- "
User.select(:name).reselect(params[:column])

Note this selects all columns from a different table:

1

SELECT * FROM orders -- FROM "users"

rewhere

rewhere is analogous to reselect but it replaces the WHERE clause.

Like where, it is very easy to open up rewhere to SQL injection.

1
2

params[:age] = "1=1) OR 1=1--"
User.where(name: "Bob").rewhere("age > #{params[:age]}")

Resulting query:

1

SELECT "users".* FROM "users" WHERE "users"."name" = ? AND (age > 1=1) OR 1=1--)

Wrapping Up

Any other new methods that allow SQL injection? Let me know!

Want to find out more?

• rails-sqli.org for the complete list.
• Brakeman to help find vulnerable queries in your code.
• OWASP SQL Injection Cheat Sheet to learn more about preventing SQL injection.

Backstory

Recently, a friend asked me if just calling constantize on user input was dangerous, even if subsequent code did not use the result:

1

params[:class].classify.constantize

Brakeman generates a “remote code execution” warning for this code:

Confidence: High
Category: Remote Code Execution
Check: UnsafeReflection
Message: Unsafe reflection method `constantize` called with parameter value
Code: params[:class].classify.constantize
File: app/controllers/users_controller.rb
Line: 7

But why? Surely just converting a string to a constant (if the constant even exists!) can’t be dangerous, right?

Coincidentally, around that same time I was looking at Ruby deserialization gadgets - in particular this one which mentions that Ruby’s Digest module will load a file based on the module name. For example, Digest::A will try to require 'digest/a':

2.7.0 :001 > require 'digest'
 => true 
2.7.0 :002 > Digest::Whatever
Traceback (most recent call last):
        5: from /home/justin/.rvm/rubies/ruby-2.7.0/bin/irb:23:in `<main>'
        4: from /home/justin/.rvm/rubies/ruby-2.7.0/bin/irb:23:in `load'
        3: from /home/justin/.rvm/rubies/ruby-2.7.0/lib/ruby/gems/2.7.0/gems/irb-1.2.1/exe/irb:11:in `<top (required)>'
        2: from (irb):2
        1: from /home/justin/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/digest.rb:16:in `const_missing'
LoadError (library not found for class Digest::Whatever -- digest/whatever)

The Digest library uses the const_missing hook to implement this functionality.

This made me wonder if constantize and const_missing could be connected, and what the consequences would be.

Constantizing in Rails

The constantize method in Rails turns a string into a constant. If the constant does not exist then a NameError will be raised.

However, it is possible to hook into the constant lookup process in Ruby by defining a const_missing method. If a constant cannot be found in a given module, and that module has const_missing defined, then const_missing will be invoked.

2.7.0 :001 > module X
2.7.0 :002 >   def self.const_missing(name)
2.7.0 :003 >     puts "You tried to load #{name.inspect}"
2.7.0 :004 >   end
2.7.0 :005 > end
 => :const_missing 
2.7.0 :006 > X::Hello
You tried to load :Hello
 => nil

If const_missing is implemented with behavior based on the constant name, such as loading a file or creating a new object, there is an opportunity for malicious behavior.

Some Vulnerable Gems

Fortunately, const_missing is not used very often. When it is, the implementation is not usually exploitable.

Searching across ~1300 gems, I found only ~40 gems with a const_missing implementation.

Of those, the majority were not exploitable because they checked the constant name against expected values or called const_get which raises an exception if the constant does not exist.

One gem, coderay, loads files based on constant names like the Digest library. Also like the Digest library, this does not appear to be exploitable because the files are limited to a single coderay directory.

The next two gems below have memory leaks, which can enable denial of service attacks through memory exhaustion.

Temple

The Temple gem is a foundational gem used by Haml, Slim, and other templating libraries.

In Temple, there is a module called Temple::Mixins::GrammarDSL that implements const_missing like this:

1
2
3

def const_missing(name)
  const_set(name, Root.new(self, name))
end

The method creates a new constant based on the given name and assigns a new object.

This is a memory leak since constants are never garbage collected. If an attacker can trigger it, they can create an unlimited number of permanent objects, using up as much memory as possible.

Unfortunately, it is easy to exploit this code.

Temple::Grammar extends Template::Mixins::GrammarDSL and is a core class for Temple. Let’s see if it is loaded by Haml, a popular templating library often used with Rails:

2.7.0 :001 > require 'haml'
 => true 
2.7.0 :002 > Temple::Grammar
 => Temple::Grammar 

Great! What happens if we try to reference a module that definitely does not exist?

2.7.0 :003 > Temple::Grammar::DefinitelyDoesNotExist
 => #<Temple::Mixins::GrammarDSL::Root:0x000055a79b011060 @grammar=Temple::Grammar, @children=[], @name=:DefinitelyDoesNotExist> 

As can be seen above, the constant is created along with a new object.

To go one step further… does the use of constantize invoke this code?

We can test by loading a Rails console for an application using Haml:

Loading development environment (Rails 6.0.3.2)
2.7.0 :001 > require 'haml'
 => false 
2.7.0 :002 > 'Temple::Grammar::DefinitelyDoesNotExist'.constantize
 => #<Temple::Mixins::GrammarDSL::Root:0x000055ba28031a50 @grammar=Temple::Grammar, @children=[], @name=:DefinitelyDoesNotExist> 

It does!

Any Ruby on Rails application using Haml or Slim that calls constantize on user input (e.g. params[:class].classify.constantize) is vulnerable to a memory leak via this method.

Restforce

A very similar code pattern is implemented in the restforce gem.

The ErrorCode module uses const_missing like this:

1
2
3
4
5

module ErrorCode
  def self.const_missing(constant_name)
    const_set constant_name, Class.new(ResponseError)
  end
end

Nearly the same, except this actually creates new classes, not just regular objects.

We can verify again:

Loading development environment (Rails 6.0.3.2)
2.7.0 :001 > require 'restforce'
 => false 
2.7.0 :002 > Restforce::ErrorCode::WhateverWeWant
 => Restforce::ErrorCode::WhateverWeWant 

This time we get as many new classes as we want.

This has been fixed in Restforce 5.0.0.

Finding and Exploiting Memory Leaks

Finding vulnerable code like this in a production application would be difficult. You would need to guess which parameters might be constantized.

Verifying that you’ve found a memory leak is a little tricky and the two memory leaks described above create very minimal objects.

From what I could estimate, a new Rule object in Temple uses about 300 bytes of memory, while a new class in Restforce was taking up almost 1,000 bytes.

Based on that and my testing, it would take 1 to 4 million requests to use just 1GB of memory.

Given that web applications are usually restarted on a regular basis and it’s not usually a big deal to kill off a process and start a new one, this does not seem particularly impactful.

However, it would be annoying and possibly harmful for smaller sites. For example, the base Heroku instance only has 512MB of memory.

Another note here: Memory leaks are not the worst outcome of an unprotected call to constantize. More likely it can trigger remote code execution. The real issue I am trying to explore here is the unexpected behavior that may be hidden in dependencies.

Conclusions

In short: Avoid using constantize in Rails applications. If you need to use it, check against an allowed set of class names before calling constantize. (Calling classify before checking is okay, though.)

Likewise for const_missing in Ruby libraries. Doing anything dynamic with the constant name (loading files, creating new objects, evaluating code, etc.) should be avoided. Ideally, check against an expected list of names and reject anything else.

In the end, this comes down to the security basics of not trusting user input and strictly validating inputs.

Edit: It seems some language I used above was a little ambiguous, so I tweaked it. Calling classify does not make the code safe - I meant calling classify is not dangerous by itself. It’s the subsequent call to constantize that is dangerous. So you can safely call classify, check against a list of allowed classes, then take the appropriate action.

A recent vulnerability report and the blog post behind it brought my attention back to the escape_javascript Ruby on Rails helper method.

Let me say it again… if you are calling `escape_javascript` or `j` in your Rails code, please don't! https://t.co/60KLEjHX3T

— Justin Collins (@presidentbeef) May 9, 2020

It’s bad form to drop blanket statements without explanation or evidence, so here it is:

Escaping HTML

Part of the danger of escape_javascript is the name and apparent relationship to html_escape.

HTML is a markup language for writing documents. Therefore, it must have a method for representing itself in text. In other words, there must be a way to encode <b> such that the browser displays <b> and does not interpret it as HTML.

As a result, HTML has a well-defined HTML encoding strategy. In the context of security and cross-site scripting, if a value output in an HTML context is HTML escaped, it is safe - the value will not be interpreted as HTML.

(See my post all about escaping!)

Escaping Javascript

On the other hand, JavaScript has no such escaping requirements or capabilities.

Therefore, the “escaping” performed by escape_javascript is limited. The vulnerability report states the method is for “escaping JavaScript string literals”.

In particular, escape_javascript is only useful in one, single context: inside JavaScript strings!

For example:

1
2
3
4

# ERb Template
<script>
  var x = '<%= escape_javascript some_value %>';
</script>

Use of escape_javascript in any other context is incorrect and dangerous!

This is and always has been dangerous (note the missing quotes):

1
2
3
4

# ERb Template
<script>
  var x = <%= escape_javascript some_value %>;
</script>

some_value could be a payload like 1; do_something_shady(); // which would result in the following HTML:

1
2
3

<script>
  var x = 1; do_something_shady(); //; 
</script>

The escape_javascript helper does not and cannot make arbitrary values inserted into JavaScript “safe” in the same way html_escape makes values safe for HTML.

CVE-2020-5267

Jesse’s post has more details, but here’s the gist: JavaScript added a new string literal. Instead of just single and double-quotes, now there are also backticks ` which support string interpolation (like Ruby!).

This meant it was simple to bypass escape_javascript and execute arbitrary JavaScript by using a backtick to break out of the string or just #{...} to execute code during interpolation.

For example, if this were our code:

1
2
3
4

# ERb Template
<script>
  var x = `<%= escape_javascript some_value %>`;
</script>

Then if some_value had a payload of `; do_something_shady(); //, the resulting HTML would be:

1
2
3

<script>
  var x = ``; do_something_shady(); //`
</script>

This is because escape_javascript was not aware of backticks for strings.

Dangers of Dynamic Code Generation

Let me say it again… using dynamic javascript under practically any circumstance is inviting trouble. It might be ok. I’d rather not have to worry about it. https://t.co/wnPy3OnkKI

— Shake, Oreo (@ndm) May 9, 2020

As I have talked about before, web applications are essentially poorly-defined compilers generating code with untrusted inputs. In the end, the server is just returning a mishmash of code for the browser to interpret.

However, directly trying to generate safe code in a Turing-complete language like JavaScript or Ruby via string manipulation is a risky game. Methods like escape_javascript make it tempting to do so because the name sounds like it will make the code safe.

If at all possible, avoid dynamic code generation!