[How to benchmark PBS?]
After providing bid month pairings and pre-defined activities to the PBS vendor, ask the vendor to perform two kinds of benchmark tests.
1. [Stress Bidding Test]
This test is the classic traditional seniority benchmark test. It uses the same kind of bids for all crewmembers. For example, everyone bids weekends off. However, this test can only benchmark the honoring of seniority.
A good PBS should honor the preferences to as many senior crewmembers as possible. In the above bidding example, the one that creates more weekends off should be a better one. Such a result may disappoint the junior crewmembers. After certain seniority bids are solved, junior crewmembers are always awarded with pairings that touch almost every weekend.
2. [Reality Bidding Test]
This test groups bids with very different preferences, and bids in each group are the same. For example, the top 20% of crewmembers bid the weekends off, and the bottom 20% bid a specific kind of pairing. This is the test that benchmarks the overall satisfaction. It mimics the practical bids in the real world when everyone submits bids with his/her own considerations.
Review and compare the awarded line for each crewmember from each group carefully. The one with the higher overall satisfaction is the one that should be considered. Satisfaction of both senior and junior crewmembers should be a key factor.
In both bidding benchmark tests, do not forget to compare the open time (the uncovered pairings), it should be none, or at least evenly distributed throughout the bid month. The more open pairings left, the more reserve crewmembers are needed.
The benchmark tests and the opinions from the crewmembers of existing customers tell the true story of how "good" a PBS is, not the vendor's own claim.