5/54 or approximately 0.092592593Â Â
There are 6^3 = 216 possible outcomes of rolling these 3 dice. Let's count the number of possible rolls that meet the criteria b < y < r, manually.
 r = 1 or 2 is obviously impossible. So let's look at r = 3 through 6.
 r = 3, y = 2, b = 1 is the only possibility for r=3. So n = 1
 r = 4, y = 3, b = {1,2}, so n = 1 + 2 = 3
 r = 4, y = 2, b = 1, so n = 3 + 1 = 4
 r = 5, y = 4, b = {1,2,3}, so n = 4 + 3 = 7
 r = 5, y = 3, b = {1,2}, so n = 7 + 2 = 9
 r = 5, y = 2, b = 1, so n = 9 + 1 = 10
Â
 And I see a pattern, for the most restrictive r, there is 1 possibility. For the next most restrictive, there's 2+1 = 3 possibilities. Then the next one is 3+2+1
= 6 possibilities. So for r = 6, there should be 4+3+2+1 = 10 possibilities.
Let's see
 r = 6, y = 5, b = {4,3,2,1}, so n = 10 + 4 = 14
 r = 6, y = 4, b = {3,2,1}, so n = 14 + 3 = 17
 r = 6, y = 3, b = {2,1}, so n = 17 + 2 = 19
 r = 6, y = 2, b = 1, so n = 19 + 1 = 20Â
 And the pattern holds. So there are 20 possible rolls that meet the desired criteria out of 216 possible rolls. So 20/216 = 5/54.