I've been involved in solving a similar problem, on the award flight search side. There are about 12,000 direct commercial routes offering award redemption, and the airline websites have different limitations: some only let you search one-way or only round-trip, or only one cabin at a time, or only give hints about available quantities. So you have to formulate the minimal set of queries that will give you complete information. And then given an origin and destination, figure out the valid routings, like you've discovered.
The other really interesting challenge with many of the airline websites are the extreme measures they go to, to block you from crawling their data. My favorite is British Airways, which runs 800 different tests to fingerprint your browser, from generating audio files to drawing to an off-screen canvas, all inside a custom VM with encrypted byte code. (Similar to how I've seen Google obfuscate the ReCaptcha code base.) Surprising to hear that Ryanair makes it so easy to access prices in comparison!
Many moons ago, I worked in the travel industry. Most airlines aren't running their own technology stack -- they are paying someone else $/search to drive their webpage. In a market of razor thin margins, paying per search for you to crawl their flight catalog is a big problem -- thus the arms race. The only way to win at this is to own the catalog, this is why Google bought ITA.
Ryanair is the kind of airline that I would suspect might run their own stack, and therefore not have the same cost structure.
The only airlines I know of running their own PSS are Emirates and Turkish. Ryanair are using Navitaire (Amadeus' low-cost subsidiary). To be honest, the cost structure is the same whether you're running it yourself or outsourcing - you don't want an absurdly high look to book ratio, or else you're just wasting server costs
I've flown with them a couple of times for leisure and once for work. Honestly find their economy food is better than most airlines' business food. The business seats weren't very good though, they're a few years behind the others
>"In a market of razor thin margins, paying per search for you to crawl their flight catalog is a big problem ..."
Can you say how access to an airline's flight catalog works and what that business model looks like? Is it pay per search for these third party search engines?
There are two different ways to connect to an airline inventory system (what you call the catalog). Either direct, by signing a contract with the airline, and building the necessary "pipe" (IATA are pushing for a standardisation of this, called NDC). Or through a global distribution system (GDS) - which is a fancy name for what used to be called computer reservation systems. There are three major systems, by decreasing importance: Amadeus, Sabre and Travleport. They will cache and distribute the inventory and fares for all client airlines, and also allow you to perform bookings. Every travel agency in the world (including online travel agencies like expedia) is connected to one or more of these GDS.
Now the pricing can be complex, a good first order approximation is that you pay a fee for access to the GDS, that varies depending on how many queries you are doing. For a traditional travel agency this fee will be very low, obviously for a major OLTA it will be big. However, for every booking that you perform, the GDS will give back a sizable amount of the distribution fee that it took (we are talking several euros per flight here). So as long as your "look to book" ratio is reasonable, the access costs will be small compared to the incentives you make.
Edit: Why are airlines pushing for the "direct" channels? Because it means they can cut out the middleman of course, the GDS. The problem is that this makes it hard for a travel agency to connect to every airline - the fixed costs of setting up pipes to every airline would be astronomical. And there will be an incentive for agencies to negotiate preferential deals with certain airlines. Both of these factors are obviously negative for the end user, travelers. It's a similar question to the whole net neutrality thing
Thank you for the wonderfully detailed response I really appreciate it. I have one follow up question to your reply which is who maintains Amadeus, Sabre and Travleport? Are these consortiums similar to MasterCard and Visa then? I'm guessing these are all big mainframe operations?
You're welcome. Amadeus, Sabre and Travelport are now independent companies. But indeed all of them were initially parts of airlines, that were then spun off.
Sabre for example was originally American Airlines' system, and Amadeus was initially created by a consortium of european airlines in reaction to the threat of american hegemony on this market. They are now far more profitable than their parents :)
They were all originally heavy mainframe operations (IBM TPF - which was quite an impressive system... in the 80s), but there is a transition to modern architectures. Amadeus have completed the transition to Linux boxes, Sabre are still ongoing. They both have some of the biggest civilian data centres in the world
In my defense, I left the industry in 2011, back when American lost $2B, and half of the major airlines were in bankruptcy proceedings. It looks like they found some margin in the past few years.
Fun fact: Carl actually implemented the Turing machine construction on slide 28 as a proof of concept and used it to do simple arithmetic (very very slowly) -- you can read the details towards the end of the talk, with screenshots.
I've been involved in solving a similar problem, on the award flight search side. There are about 12,000 direct commercial routes offering award redemption, and the airline websites have different limitations: some only let you search one-way or only round-trip, or only one cabin at a time, or only give hints about available quantities. So you have to formulate the minimal set of queries that will give you complete information. And then given an origin and destination, figure out the valid routings, like you've discovered.
The other really interesting challenge with many of the airline websites are the extreme measures they go to, to block you from crawling their data. My favorite is British Airways, which runs 800 different tests to fingerprint your browser, from generating audio files to drawing to an off-screen canvas, all inside a custom VM with encrypted byte code. (Similar to how I've seen Google obfuscate the ReCaptcha code base.) Surprising to hear that Ryanair makes it so easy to access prices in comparison!