Lua Script in Redis for Supporting Inventory Management System

Posted on by admin

Important Aspects In Inventory Management System

Inventory management system is a system to determine stock inventory at a certain time. Inventory management is one of the basic problems in almost every company. There are some aspects that needs to considered in inventory management system:

  • Stock control
    • reduce stock when there is any claim request
    • restock when there is any declaim request or rollback
  • Security
    • only legitimate user that able to update the stock
    • request should be idempotent, no claim/declaim operation are executed more than one in the same requests or it can produce over booking/over budget.
  • Inventory Tracking
    • contains history or transaction status for business needs or just for debugging purposes

Designing robust and reliable system also ensure high load requests (ex: flash sale or seasonal sale) are executed without any problem. Since this topic contains comprehensive discussion, this article is only focused in stock control. How do we reduce the stock or restock it.

Stock Control System

In stock control system with limited stock usually it is conducted with the following steps:

  • get current stock for particular item
  • validate total stock (current stock + requested stock)
  • claim / declaim stock based on the validation result

There are some approaches in order to overcome this thing:

  1. database procedure (with if else statement)
  2. lua script in redis (with if else statement)
  3. combination between database query and if else statement in app level
  4. combination between redis query and if else statement in app level

The benefit of number 1 and 2 is that the request is executed in one call to database/redis. It means, getting current stock, validation, and claiming are conducted in one primitive script, and claiming process can be decoupled from business logic. However it needs more effort to debugging script.

Number 3 needs to be executed in transaction mode, otherwise it is hard to rollback. This approach also have multiple db calls. Number 4 is hard to implement since there is no rollback mechanism in redis, we need to create function to rollback the request manually. Therefore, number 3 and 4 can produce dirty data when the revert failing.

See also  Key-Value Store in NATS: Simple State Management

Number 2 and 4 also have another drawback, since redis does not provide real persistent data. In RDB (Redis Database), it snapshots the db for certain time, while in AOF (Append Only File) it can slowdown your process depending on the exact fsync policy. However if fsync disabled it should be exactly as fast as RDB even under high load.

There is an experiement from other team in my company, they give up to use database (postgresql) in claiming process and move to redis because it turns out that using redis in voucher claim system is faster than db call because of redis call is cheaper than db call.

Lua Script in Redis

Prerequisites

– Redis 2.6.0 or above

Lua is a powerful and fast programming language that is easy to learn and use and to embed into your application. Lua is designed to be a lightweight embeddable scripting language. It is used for all sorts of applications from games to web applications and image processing. See https://www.lua.org/start.html for more detail.

Redis Pipeline vs Redis Transaction vs Lua Script in Redis

Maybe you think that Lua script in redis is similar to redis pipeline and redis transaction. However lua script is used for different use case. Pipelining is primarily a network optimization. It means the client collects some commands and ships them to server in one call. But it is not guaranteed to be executed in a transaction, it can be overlapped with other commands that run at near the same times. By shipping commands in one call, it is saving round trip time. In other side, redis transaction is used to ensure there is no overlapping commands during execution.

Most commands in redis pipeline and transaction can be implemented in Lua script. In Lua script we can design flow by using if else condition or looping, because basically it is programming language and redis command is called within the Lua code. Similar with redis transaction, redis will block other operations.

Eval Command

By using EVAL, we can evaluate scripts using the Lua interpreter. Below is the structure of EVAL command:

EVAL {lua script} {number of keys} {key 1} {key 2 (if any) and so on} {arg 1 (if any)} {arg 2 (if any)}
  • First argument is lua script
  • Second argument is number of keys
  • The next arguments after the second argument are the keys. If number of keys 0 then the next argument is script argument (if any)
See also  Key-Value Store in NATS: Simple State Management

Example:

EVAL "return {KEYS[1],KEYS[2],{ARGV[1],'hello world!'}}" 2 first_key second_key first_argument

It returns:

  1. first_key
  2. second_key
  3. 1) first_argument
    2) hello world!

As you can see, array result is started from 1 not 0 and it will return Lua array as Redis multi bulk replies. Then, in order to call redis we can use these 2 commands:

  • redis.call
  • redis.pcall

Command redis.pcall is similar to redis.call, the only different is redis.call will return Lua error while redis.pcall will trap the error. Please find example below:

127.0.0.1:6379> hset foo bar 1
(integer) 1
127.0.0.1:6379> eval "return redis.call('get','foo')" 0
(error) ERR Error running script (call to f_6b1bf486c81ceb7edf3c093f4c48582e38c0e791): @user_script:1: WRONGTYPE Operation against a key holding the wrong kind of value
127.0.0.1:6379> eval "return redis.pcall('get','foo')" 0
(error) WRONGTYPE Operation against a key holding the wrong kind of value

Currious what happen if we monitor it? lets see what is actual happen if we execute this command:

127.0.0.1:6379> eval "redis.call('set','foo','bar'); return redis.call('get','foo');" 0
"bar"

Then in monitor we can see this result:

~ redis-cli monitor
OK
1618785957.379546 [0 127.0.0.1:53499] "eval" "redis.call('set','foo','bar'); return redis.call('get','foo');" "0"
1618785957.379588 [0 lua] "set" "foo" "bar"
1618785957.379601 [0 lua] "get" "foo"

Nothing surprising, isn’t it? Yes, redis commands are executed sequentially.

Use Case

Now, lets move to the use case. Lets say we need to create simple counter for claimed voucher and each voucher has limited quantity. If user claims a voucher, then we need to increase its counter and can not more than the voucher quantity. If user declaims a voucher, then we need to decrease its counter and can not less than 0. Please take a look this Lua script:

local TOTAL_CLAIMED_QTY = tonumber(redis.call('incrby',KEYS[1],ARGV[1]));
local status = 1;
if TOTAL_CLAIMED_QTY > tonumber(ARGV[2]) then
	status = 0;
	redis.call('decrby',KEYS[1],ARGV[1]);
elseif TOTAL_CLAIMED_QTY < 0 then
	status = -1;
	redis.call('set',KEYS[1],(TOTAL_CLAIMED_QTY-tonumber(ARGV[1])));
end;
return status;

Let say we assign this script into variable lua_script and the key is counter_key. Also client needs to know the result of the command, so that we need to return status. In this case, 1 is success, 0 is exceeded, -1 is invalid (if counter less than 0). As you can see we can describe variable with local and since redis return string by default then we need to cast it by using tonumber.

Then we execute it by calling this command:

EVAL lua_script 1 counter_key {increment} {quantity}

First we need to increment KEYS[1] with ARGV[1]. Please note that negative ARGV[1] is equal to decrement. Then Lua will check 2 conditions, exceeded condition and invalid condition. Those, we will have 3 scenarios as described below:

  • Scenario 1
    If current counter + increment is less than or equal to max allowed quantity or current counter + decrement (ARGV[1] is negative number) is more than or equal 0 then EVAL will return 1.
  • Scenario 2
    If current counter + increment is more than max allowed quantity then revert it by using decrby and return 0.
  • Scenario 3
    If current counter + decrement (ARGV[1] is negative number) is less than 0 then EVAL will return -1.
See also  Golang: Benchmarking and Optimizing Code

We can also add expiry to the key before returning the status by using this command:

redis.call('expire',KEYS[1],ARGV[3]);

So that we need to add another argument to specify TTL. Another approach is that we can check the quantity first then increment/decrement when remaining quantity is sufficient.

local CURRENT_QTY = tonumber(redis.call('get',KEYS[1])) or 0;
local status = 1;
if CURRENT_QTY > 0 then
	if CURRENT_QTY + ARGV[1] <= tonumber(ARGV[2]) then
		redis.call('incrby',KEYS[1],ARGV[1]);
	else
		status = 0;
	end;
else
	if CURRENT_QTY + ARGV[1] > 0 then
		redis.call('incrby',KEYS[1],ARGV[1]);
	else
		status = -1;
	end;
end
return status;

The benefit of second approach, we do not need to revert back when quantity is exceeded or less than 0, while the drawback of using the second approach, we need to call redis twice when remaining quantity is sufficient. It depends of the cost that you want to pay.

Conclusion

By using EVAL, we can execute Lua script and call redis command within the Lua code. Redis will block other operations and ensure there is no overlapped executed commands. Thus, we can design flow in Lua script it self. There are a lot of redis command that can be used in Lua script, please refer to this page.

Category: Lua, Programming, Redis

Leave a Reply

Your email address will not be published. Required fields are marked *