Connect Working Group
23rd May 2024
11 a.m.:
REMCO VAN MOOK: It is almost 11 o'clock, and we would like to have a timely departure, so if could you all please go and find your seat. We still have some passengers arriving so we will leave the doors open for a little bit longer. All right. It is just after 11 a.m. on Thursday morning here in the side room, and that means welcome to your second most favourite Working Group of this week, Connect Working Group, as always we appreciate you using this service, this flight has four emergency exits, four in the back, I don't know about that configuration, please do not use the windows as emergency exits, that can turn nasty.
I will be your cabin attendant today together with my lovely colleagues Florence and Will, we will be taking care of you for this session. A short reminder, please keep your seatbelt fastened at all times in case of an unexpected Code of Conduct, do not forget to rate the talks and have yourself a seat. This is a short flight so there will be no drinks service, sorry. But there will be drinks and food after landing, so you are going to have to suck it up a little bit.
Now, okay, Will, what have you done?
That is not a touch screen, that's fantastic. Here we are. So welcome to Connect Working Group. This is the agenda for today, Will ‑‑ we talked about this. Okay.
This is better. Good. So, opening, well, I think, what I equal fives as an appointment. We have Martin from the RIPE NCC over there who has gracefully volunteered to be a scribe, that's the way that works, Martin?
SPEAKER: I was volunteered
REMCO VAN MOOK: You were volunteered. This is the agenda, we will have an open session as we usually do. A couple of housekeeping notes. You have all read the minutes, right? So, who else spotted the typo and where was it? No the agenda, the minutes from last time. So, I'm happy to announce that Mirjam Kuhne gets the gold star for the Chair from pointing it was written as RIDB which I am sure you have all noticed but you were too modest to say anything about. With that change, I would like to accept the minutes.
Then, next up, we have the Chair selection process. And these are some fantastic slides so I am going to have Will talk about this.
WILL VAN GULIK: Absolutely, thank you. So, basically, that's the process that we got in the past. We mentioned that already at RIPE 75 and RIPE 87, so that's a short reminder. So we launched the process to request for new Chairs and so what's happening this time is that Florence and Remco are stepping down and I'm reaching two term also. Therefore, I have got some ‑‑ something for you.
REMCO VAN MOOK: Is it chocolate?
WILL VAN GULIK: No. T‑shirt. One for you, it's the other way around. I had a small problem with the back but in the front it's written "I was a Connect Working Group ‑‑ cabin for ten years and all I got is this lousy T‑shirt".
(Applause)
And as usually when we call for nomination, we didn't add anything in our timing, including me. But we have three late nominations, and so like in order that the Chair receive them that I ‑‑ we got nomination from Stavros and Paul and as it was after the time, well we accepted them and we thank them for this new endeavour and so they will be the two new Chairs going on with me for the next term. So thank you to ‑‑ for them for that and you will see them with me in Prague, but for the rest now, we can go back to our agenda.
So, the Chair selection process is done, and I think we can go on with Stavros with the a BCP for the use of IRR DBs by IXP on route servers.
STAVROS KONSTANTARAS: Thank you for the introduction, this is a follow‑up presentation and something that we already discussed for the last two RIPE meetings.
So, quick recap for the BCP that we proposed, is actually that IXP operators must trust and use the following databases in order to build and maintain root server filters we are talking about five big RIR databases together with the official delegations.
And because we understand that this cannot be implemented in one day, one month and will take some time, we introduce what we call the grace period and during this grace period actually we give time to our members and customers to do their clean‑up job and because we don't want to create issues on the routing ecosystem or lose traffic we actually extend the list of the five RIRs with those four that you can see in red, RADB, NTT and Level 3 so we allow some time to customers and members to do their job and during the grace period, actually, as you can see ‑‑ at the end of the grace period, we drop the support of those four unofficial databases and during the time of the grace period, the IXPs must do their best in order to inform the customers about what is coming and what we are going to achieve.
But then the question starting coming up, what is the impact if we apply this BCP tomorrow? We know the theoretical impact. And that is, as I said, there's going to be a huge move of objects, route num, Route‑6 that they need to move back to the official databases and customers need to go and update jobs in the official databases. However, we thought ‑‑ we spotted that global operations might be affected because of the ARIN legacy space and I am coming back to that a bit later but, so then we started focusing, the authors in the ARIN legacy space and questioning ourselves like okay, how many prefixes will be affected because of BCP and what type of prefixes might get lost?
So, we did a practical analysis, my friend Marco wrote a nice script, quick and beautiful, that actually you fill the script with your BIRD route table and also with the ARIN legacy space that is registered in server for ARIN and you actually do the script and you get the result. And so what we discovered, I can say for AMS‑IX we discovered approximately 5,580 that prefixes in the route servers belong to the ARIN legacy space of the full table, Interlan around 4.3K of those prefixes, minap 3.9, they took a little extra time and did their checks and they will contribute a little bit later in presentation with the results.
I started looking into this prefixes, I could see that 3.3 of them come only from one member, who was local customer, actually he was doing local peering and then the other customers were contributed a little bit less amounts of ARIN legacy prefixes.
Going back into the database distribution of those legacy prefixes, we could see that the majority of them, 4,200 actually, were registered in RIPE DB, what a surprise.
Yeah, validity. That was also not a surprise. I could see that approximately 60% of those ARIN legacy prefixes are classified as either invalid in our root servers, that means that they were coming from an AS cone that was not correctly configured or allowed in the route service of AMS‑IX, of course talking about RPKI, almost all of them were invalid, right? It's ARIN legacy space.
Origin, yes, nice. Looking into the origin of those prefixes, we could see that almost 2000 of them had, they had AS 174 as an origin. We can do now Whois and find out who is AS 174. And then the other prefixes had just ‑‑ I see surprise with AS 174, yes.
The problem with ARIN. So what we say legacy resources, this is what we are talking about, so legacy resources are the networks allocated before ARIN was established in 1997. Unlike the other RIRs, ARIN has decided not to provide IRR and RPKI services to legacy resource holders. And actually there's no authoritative IRR server for these IP networks so they cannot be validated using RPKI, either.
So, what we see in Europe about those legacy resource holders, we could see that again looking into the root server prefixes that most of them are access operators that networks from cogent. We saw a lot of US universities, a lot of US local and central government services, some enterprise networks and the local branches of them in Europe, amateur packet radio that we mentioned in the previous RIPE meeting and the managed DNS root servers, but also what we saw and that was a not surprise for us, that a lot of those prefixes belong actually to Akamai, to the DoS scrubbing centre and so that was something we had to talk with Akamai people during the RIPE meeting.
So then if we try to screen those resources now in our root server prefixes we can see that AMS‑IX, yeah, from all those services ‑‑ all those prefixes you have 214 K, 1400 actually be long to cogent and 93 US educational institutes, we had one prefix from US Government, 61 prefixes from amateur packet radio and 456 other. And I am mentioning AMS‑IX because these are the biggest numbers, low NAP has a little bit less as you can see, you can start the study the numbers later. Checking again the classification of those prefixes, as you can see majority of them come from cogent space, 430 from prefixes and advertised by Hurricane Electric and belong to US educational institutes, Hurricane Electric also advertise 293 US military prefixes to us, 107 from government, 56 from amateur packet radio and so on.
So, conclusions:
What we believe for this BCP is it enhances the routing security and helps towards this direction and we have very strong signs that not supporting any more filtering with RADB is feasible but annoying.
Losing the peering routes of US‑based networks is not an issue because there's not much traffic going there anyway. The major casualties that could be impacted are the local networks that list cogent IP space and root server.
We had collaboration with DE‑CIX that they did analysis and due diligence and we will present the results.
Matthias: When Stavros approached us about that we thought it might be good to have the traffic impact of this and that's what we did.
So, we used our router to try to simulate three scenarios, the current scenario where we are using default IRR DB search that we have in production. And we allow our members to prioritise their preferred database to the front of the search order. We have a scenario where we looked at RADB and IRRs that means the default search order we have now without non‑RIR DBs but with RADB to see the impact on the traffic. And last we have the RIR only where we only look at the official databases that means excluding RADB.
So we took these three scenarios and we pushed them through our root server and came up with three different root server configurations for the scenarios, and we repassed these filters from the configurations and looked at what amount of allowed prefixes you have in these three scenarios, and from there on we can compare how many prefixes do you lose from one scenario to another and what we also did we took at flows from Frankfurt and how much traffic is in these prefixes, because that gives you an estimated traffic loss.
So, looking at the AS filter/prefix loss we had in the configuration, you immediately see on this slide that if you do not exclude RADB from the filter generation you don't lose very much, that's the orange to the green line so essentially we are using one to two percent of the filtering list we have there in terms of /24 coverage. It's about the same amount, right? We are losing 2% of /24 coverage.
However, if you go down to the RIR only scenario you are losing quite a bit so roughly one‑third of the filtering lists and roughly 12% of /24 coverage in the overall IRR filters.
And last but not least, because we are running out of time, we also looked at the traffic that is in these prefixes and on the left side you see an hourly bins, that traffic loss that we see in the scenario that is not excluding RADB from filter generation and you see there that the loss is really close to negligible, we are talking about half a gigabit or something like this and on the right side you see it if you exclude RADB from the filter list so that's a lot more so that has quite some impact. You see in worst case during this specific we could have potentially lost 250 gigabits, however it's important to understand this is an upper bound, because we don't see whether this is steered by private peering or by the root server so worst case scenario.
Our conclusions phasing out alternative RADBs is very, very doable, but RADB has substantial list and the traffic loss could be substantial so be careful with that. And our proposals are going forward with this BCOP is together like‑minded IXPs and find a date for removing the non‑official databases somewhere towards the end of the year or beginning of next year. I think it's very doable except RADB and we should coordinate and prepare customer communications, monitor the result and then implement the changes at date X and I would postpone the RADB issue because I really think it has substantial impact and we need to see how we can handle RADB afterwards. That's all.
FLORENCE LAVROFF: Thank you very much. It's time to take questions from online. I could not see any questions so far. But we have question from the floor. Marco.
AUDIENCE SPEAKER: Hinge your last analysis is ‑‑ because you measured what will happen but not using RADB but what we are actually interested in is what will happen by not using the ‑‑ by ignoring the legacy resources. We have, I think it is totally reasonable to assume that no legacy ‑‑ that currently used RADB will reduce the routes in the authoritative IRRs. Marco...
Matthias: This analysis is what happens if we do it tomorrow but I guess it's roughly equivalent to say what you are saying with the legacy space holders because it's mainly that and that's the impact on them you are seeing here, right? However you need to make sure that they move to the authoritative IRRs and I think that will take some time.
MARCO: The point is that legacy space holders cannot move but I believe that there is still a significant number of US networks that do it, they are just use it to use only RADB and not the ARIN database but they could move.
Matthias: If they are willing to, yes.
AUDIENCE SPEAKER: Liberty Global. Your research actually matches my experience, so I want to say that it's good to do. One word of advice, especially, we are in the room mostly with, well operators, right? There are a number of sponsors that I see on the platform that actually use RADB as a standard ‑‑ standard database to create an object, one suggestion is go and talk to them to stop doing that. And use the official ones so especially I see the problem mostly with least addresses
MATTHIAS: I can take that. Yes, they do, yes they do, and the question is, why they do it? And we believe they do it mostly because of easeiness. If you are talking about good space, registered space that can be ‑‑ can be registered in official database then the question is why they use RADB?
AUDIENCE SPEAKER: Most of them do it because it's automated that way a few years ago so we could start with that ‑‑
MATTHIAS: Exactly. Nowadays I was speaking during the RIPE meeting with RIPE meeting, ARIN people, LACNIC, most of them have tools to make this process easier as well so the information currently is not valid, a valid reason. As I said we introduce a grace period that allows people to implement tools and start slowly moving the objects to the official database because now we have LACNIC, we have AFRINIC, there are automation tools around. I mean, excuses are not valid any more. Only legacy space is valid excuse.
AUDIENCE SPEAKER: Perhaps, I support the proposal and I have already started connecting ‑‑ contacting our customers to stop using a year ago
Matthias: We thank you very much for that.
AUDIENCE SPEAKER: I am affiliated with DE‑CIX SwissIX, and my question goes more in this direction: As far as I see you already ‑‑ you addressed it already at the RIPE meeting in Rome, I haven't seen so, far, that this is a coordinated approach with all of the IXs around the world. What is your plan to do that? I haven't even seen, we have a global mailing list for IXs and I haven't even seen this topic popping up there
MATTHIAS: Thank you for bringing this up. There should have been a slide, however. So the plan, what we hope so far, AMS‑IX, Marco can and DE‑CIX people to see the impact, are we doing something super crazy that will destroy operations globally? I think this is not the case. So the plan now is we are going to bring this topic on the mailing list and do a call of adoption, it will be ‑‑ the BCP will go to the right website, will become an official document and we have already started discussing with France /EUBGS, luck /EUBGS, all the supporters, the first adopters to set up a date where all together we enter into phase 1 and this discussion will come immediately after the RIPE meeting, we are going to start discussing this starting date and we are going to announce it once the BCP is adopted by the Chairs of the Connect Working Group.
AUDIENCE SPEAKER: Second is, I see that RADB has had the impact and what you can't do and say okay, there are databases which can authorise but there is one who does not but if I do that, that hurts me. So, I'm not willing to cut off all because that hurts me and this sounds a little bit arbitrary, and a little bit up up fair, at least in my opinion. (Unfair)
MATTHIAS: We will see with the other colleagues what will happen with RADB. For now, I checked personally a little bit the an impact and so far ‑‑ I saw for example, from all the objects we used to configure them, approximately 30% are duplicates, means that these objects exist in official databases but that's perhaps a little bit older. The last motified date was older from what we see in RADB. So 30% can easily be moved to official database and we are going to start digging further into the remaining 70% because we believe a lot of people use it because of easeiness and not because there are serious issues and let's not forget RIPE has a policy that allow operators to bring legacy space for free into the RIPE database and can be validated and can be the RIPE database and can also have RPKI. So, those policies can happen in the direction to go from RADB. We will evaluate of course during the year with DE‑CIX and other colleagues from France‑IX, Interlan, how things progress, right? But I cannot tell you for now how we are going to be.
AUDIENCE SPEAKER: Thank you.
ROB EVANS: Which is the UK network. Internet 2 which is the UK network has quite an active securitying routing security Working Group which a lot of US people are involved in, I am going to send a copy of this presentation to that list and maybe they can help chip in and see what they can do
MATTHIAS: Please do, yes, that would be nice, thank you.
FLORENCE LAVROFF: Last question.
REMCO VAN MOOK: Speaking as the ‑‑ as ‑‑ for asteroid, an internet exchange company, I have two points. First of all, I think the information you have here based on the European exchanges is very valuable. From personal experiences in other continents I can tell you that messing around with RADB for IRR filtering will break things quite severely, much more than than in Europe.
The second is, I mean, another speaker referred to the kind people who are sponsoring this meeting, who are all frequent customers of RADB. I think we should ask ourselves the question: Why do they prefer RADB? Is there a trick we are missing on the RIR databases? Because RADB is a paid service, they pay 525 ‑‑
Matthias: 425 dollars
REMCO VAN MOOK: You are not for profit, you get the discount. 525 a year and then there's a support phone number, an e‑mail address, which companies love and is that maybe something that we should be fixing? Is that something that the IRRs should be looking at?
Matthias: We can transfer this question to RIPE personnel, maybe Maarten can take a note of that. Yes.
FLORENCE LAVROFF: Last question.
AUDIENCE SPEAKER: Thank you for allowing know speak, I know I came late to the microphone. Thank you for the work you are doing, I am from ARIN, my name is Richard gem son and we share your pain with the legacy space, quite frankly. I did want to say there has been significant efforts that have shrank the legacy non‑contracted space over the years. One of the big ones of course is the transfer market; we are seeing a lot of the legacy space come under contracts with RIPE NCC, APNIC, inter on and LACNIC over the years. The United States Government and Internet 2 have also gone through significant efforts over the last two years and they are shrinking the amount of space that is considered legacy, not under contract. When these organisations do sign contracts one of the regional registries they can come under an authoritative Internet routing registry and that's a bonus for everyone, that is a good thing. But the legacy problem will not go ahead completely, unfortunately, and I wanted to share say we share your pain on that. We are restricted as ARIN on what we can do with that legacy IPv4 address space and we are locked down by policy and the wishes of the community on that. There may be additional discussions over the coming years that can seen see that shrink more but I just wanted to say I appreciate the work that everyone is doing here and we share your pain
MATTHIAS: I would like to add one last comment on that and we can close it. Yes, the ARIN legacy space, we know that's what we are here, we try to do the impact analysis and so on, but as we said the authors, we believe that, okay, in the previous RIPE meeting also in Rome and Rotterdam when there was this presented, there was a huge support also from global operators and they said yeah, it's a great idea, let's do the BCP, IETF and start applying global ‑‑ hold on, hold on, we have ARIN legacy space, there are some other issues, you cannot apply it globally and some regions like yours are not ready yet so the idea for us is not to make it in the RIPE region because it's quite clean and that can be implemented easily, see how it goes, and see how the case progresses with ARIN legacy space and if you guys manage to do it in quite quickly and successfully, then perhaps can become a global BCP or something.
AUDIENCE SPEAKER: I want to add, there should be no expectation that the legacy space will go away as a problem any time in the next few years. There will always be a large amount of IPv4 addresses there and eventually the policy community may ask the registry to do something more impactful, to clean that up, but at this point, the efforts that I described at the microphone are the ones that have been allowed to this day
MATTHIAS: Right.
FLORENCE LAVROFF: Thank you, I think with that we can close questions queue now. Thanks a lot.
(Applause)
Please rate the presentations, it's important. And we have now our next talk, Andrei Robachevsky talking about the business case for routing security. Thank you.
ANDREI ROBACHEVSKY: Thank you. Global sign he will alliance, that's the new home for the the MANRS secretariat as of this year. I would like to update you on some efforts we are doing at MANRS on looking how can we encourage better adoption of stronger securitying controls and with greater level of assurance that those controls are implemented.
But before I start with this business case and possibility of business case, I would like to show a very short update on why routing security is important and probably, I will skip through the slides quickly. But the thing is routing security is very important because it's foundational thing and we know that many of the attacks happen on the higher level they use routing vulnerability securities, it's not traffic from A to B, but it's also higher level DNS or this particular case, for instance, where software repository was affected, right, like software was malicious software was impersonated by hijacking a software repository which led to significant monetary impact.
So, routing system is important because it's foundational.
But although we know about vulnerabilities and have the tools for many decades, not one, maybe two decades, if not more, we still are at the phase where there is a problem, a big problem to solve, and why is that? Because what we face here is so‑called collection action problem, the thing is the actions do to support routing security, they benefit more your neighbours, other networks that operate in the Internet than yourself, and we all understand that if all of us come together and implement routing ‑‑ securitying routing controls, make it more secure it's better for everyone but individual actions, they do not have a lot of incentive.
So, well, MANRS was created almost ten years ago as a response to ‑‑ or as an approach to motivate this collective action to bring operators together to increase adoption of routing security practices. And while it has many facets, but simplified form, it really is done two pillars, the one is baseline, we need to reference something which is norm. As you probably know, MANRS is not a best current practice. It's based on sort of a distilled from best current practices, it doesn't create a new one, it actually creates a baseline, security baseline that every network should be able and probably should implement. But baseline, the recommends is not enough to motivate the action. Another thing is that you grow the community of committed operators that can demonstrate that they really are committed to that baseline by, you know, transparency. That's why basically in a nutshell, what the MANRS is.
But, all this collective action and securing the comments, it has limitations, and the limitations that the business case, individual business case, remains quite low, quite weak. Of course, if you implement those controls I think you can tweak it to your own network security, for instance, you secure your customers do not become your providers, for instance, or some other things. But overall it doesn't deliver a strong business case because if others do not implement those routing security controls you are still very unsecure, right?
Another thing that because the business case is not very strong, our attempts to maybe make routing security requirements more strict or maybe require more cooperation from MANRS participants, not always succeed, because it should be matched with sort of increased value proposition that MANRS deliver. Right now, MANRS deliver value proposition that you can demonstrate, social responsibility, this kind of stuff. So, we are wondering how can we achieve a better sort of improve this on the business case? And while MANRS looks at the comments, another effort which, I will introduce called MANRS+, that's working title, is looking at business relationship and business relationship between enterprise customers and their connectivity providers and I use enterprise customers in a very broad sense; it's also universities, you know, other types of organisations, sort of edge networks that would be better maybe phrasing of that.
And another problem with routing is that one provider cannot solve the routing security problem, right? Even if your provider is sort of secure from routing point of view, there are other operators that contribute to security ‑‑ insecurity of the routing system so you can't guarantee as network operator that you will deliver traffic for your customer without interruption because other things may happen elsewhere, which is not completely true for enterprises. If you look for enterprises and look at their past to their digital assets outside such as clouds, private Cloud or look at their business to business relationship, the Internet is very small. So we ran some experiments, we ran a few experiments, a few scenarios, we looked at the Australian banking sector and looked at where they can store some of the digital assets, not saying that they do but just imagine and the path between them and those sort of Cloud CDN providers is very short, it's just one, maximum two, intermediaries. By the way, the colours here, they indicate MANRS readiness, that's what we call MANRS readiness, you can see on the MANRS participants table. For instance if you can see that all the intermediaries on this path, all, meaning one or two, MANRS ready, that gives you certain assurance that your traffic on this path is more or less secure.
We also looked at another scenario and this is automotive industry and we will look at one side sort of major manufacturers and then their mainlier suppliers and again, the thing is here, you see more intermediaries on the path but the point here is if your partner and yourself choose provider wisely, then you can have higher level of assurance that your traffic will be safe, will not be hijacked or leaked and will be delivered.
So, in fact, if you look from an enterprise point of view, routing security is nothing more and nothing less than supply chain security. It also allows us to make routing security better understood by security professionals. We here as operators understand routing security very well but if you go to cyst or cooperate security, especially in enterprises they will not really see how this concept and how those practices feed in their security framework and their security organisation. Now, if you say well, this is part of your supply chain security things become more clear, and priorities become more clear. So, in fact, if enterprise starts demanding those routing security controls and certain level of conformance from their connect provider or start using those things in their selection of connectivity provider, that creates a high business case for those providers to implement those controls and therefore get competitive advantage.
So, basically, what we are trying to sort ‑‑ where we are going and trying to approach some certificate, outside the telecom industry such as banking or healthcare or automotive is your connectivity or Cloud provider the first line of defence? Which it probably is or could be. Or the weakest link?
So, the problem with security, one of the problems with security, is the signalling mechanism, so how a party such as enterprise can figure out if their provider is really implementing those security controls? That's why all this kind of things like certification are developed. This is a way for you to communicate your security to relying parties that can make informed decisions.
Now, MANRS as such, you can see MANRS as very soft certification of reputational service. As I said, the problem with MANRS, although we
AUDIENCE SPEAKER: Networks and we measure networks on continuous basis on the performance (audit) ‑‑ has caveats, it cannot measure with the really high level of assurance because if you want to go that way you require cooperation from your ‑‑ from the networks you measure, right? Maybe you ask your networks to run certain experiments, maybe you ask networks to establish peering relationship with you so you can test something there, so basically, the current framework is passive and to get higher level of assurance you need an active measurement framework that requires more cooperation.
So MANRS is having this in mind, a Working Group was created in MANRS that we are working on developing those controls, starting with the baseline of MANRS but maybe making them stronger, right? And also, looking from a point of view not what is possible to implement or it is visible to implement but how can can we assure that those requirements, that those requirements are implemented?
Okay. So, those are the areas what we call security domains are MANRS+. Very similar to MANRS, with some additions, slightly extend the scope you can see DDoS mitigation and security services are there, I will not go into details, they are questions I can answer.
Well, this is ‑‑ this is just to show, this is not ‑‑ we also have a spreadsheet with those controls there. Now, most of the informations in the auditing requirements, because they dictate what the controls are, it doesn't make sense to recommend some control if it cannot measure that this control is implemented. That's the main premise.
Measurements. We are thinking of enhancing, making more active measurements and interacting with those parties that are being audited because on side auditors is ruled out of the picture, what we need to do in continuous automated audit in MANRS+ effort.
That's the current effort. So we release the draft document, you can find us on the website, we are working on, basically finalising the diversion of the control metrics for the MANRS+. We haven't discussed how this is going to be deployed and what kind of shape it could take. It's quite possible some of those things will propagate to the baseline MANRS and it's still ongoing so if you are interested, let us know, I will be happy to see you and to see your contribution in the Working Group. Thank you.
(Applause)
WILL VAN GULIK: I don't see any question, I don't think we can had in the queue, thank you very much.
Now, Paul is going to give you a small Peering DB update.
PAUL HOOGSTEDER: Hello, all. I have been volunteered by Peering DB to give a small update, we are running out of time so I am keeping it quite short.
I am going to talk about people who have been elected or started doing work in Peering DB, and I am giving a short update about the search functions in the website, geolocation and the Beta website.
The Board of Peering DB is elected and Job Snijders and Aaron Hughes have been re‑elected and most people voted to have term limits.
The product committee are the people who discuss and decide about new features in Peering DB, both the website and the API. We have got a new Chair, jack car Oz /STKPWHROE who replaced Stephen McManus.
Then there's the admin committee and they are actually the people who work as the help desk, they are the people who answer your questions about Peering DB and also give input to the product committee about what new features might be needed or what changes need to be made because they have the most experience with the product and how people use it.
Search. One of the features which I personally use a lot is seeing things like which facilities, which data centres, are in a certain town or which exchanges operate in a certain country. Now you could do through the advanced search function, but nowadays you can also do that in text. I don't know if it's readable on the screen but search on the top I think says show me the IXs in PL, Poland. So you can see just type that and immediately get the data which you want. And it uses the ISO 3166 country codes, so PL is Poland in this example.
And divisions of countries will also be introduced at a later time.
There's a lot of geolocation data in Peering DB nowadays, especially for the facilities, which you can export, which you can import into GoogleMaps, which is very handy, as you can overlay that with KMZ you get from your Dark Fiber providers or way providers, whatever.
The Beta website. You can ‑‑ instead of using the normal Peering DB website you can use the Beta website which contains newer functions. We really urge you to try that one out. And if you have got questions, contact the product committee.
We got a request from Stavros for new functionality. We really need some more inputs from not just from the exchanges, because we already got quite a few replies from them, but also from networks that use Peering DB, to decide how much it will benefit them. Please reach out to the product committee for this.
And last I would like to ask for volunteers, volunteers especially for the admin committee, we can use more people to help with the questions which we receive and also to give input on new features that might be needed.
Thank you.
(Applause)
If there aren't any questions, then I will hand the mic over to
WILL VAN GULIK: Thank you very much for that, I don't see any questions so I think now it's the time for ‑‑ oh, we can do that. Can we have the slide for the closer, it's like two seconds ‑‑ Cams, are you here? I guess not.
REMCO VAN MOOK: I am not Bijal. She has way better hair than I do.
So, I am not Cams. Cams asked me if it was okay to do a quick announcement on the upcoming peering Asia forum, which is in November in Jakarta, apparently, so if you are interested, go look at the website and that's it.
And I wanted to get that out of the way for ‑‑ before we end up with our last topic which is going to be a discussion, so I want to have this done before then.
With that said, it's now over to Bijal, thank you.
BIJAL SANGHANI: Hello, everyone. It's good to be back here at the Connect Working Group. Something a lit different today, I am going to give a very quick update on a couple of the activities that Euro‑IX has been involved with in the last ‑‑ well, since Rome, and then we are going to have a little panel discussion.
So, first of all, who doesn't know what Euro‑IX is? Okay. So we are an association of IXPs and here you can see all our lovely members, the newest member we have is one IX which is the IXP of Ukraine, so they joined us this year. We also have patrons and the patrons are typically organisations that work closely with internet exchange points so you can see here we have data centres, vendors, and also associations like ISOC who work closely are ISPs.
What do we do? We have two events a year, we just finished our 40th forum last week which was in beautiful island of Crete, that was hosted by our member G RIX, and it was a great meeting, we had about 115 attendees, which is a lot more than we ‑‑ it's the largest number of attendees we have had since after Covid.
What else do we do? We have an ISP database that we manage, Peering Toolbox fellowship and mentor, so for ISPs who are looking to attend the forums they can apply to a fellowship. We have a mentor IS programme
PAWEL FOREMSKI: Those looking for some kind of support within their /S*P so if you know any send them over to me. (.
We also released a film last year and we have a number of Working Groups so that's kind of overall what we do and I am going to focus on a few highlights since the last meeting where I gave an update in Rome.
So, we have a Peering Toolbox and this is a website‑focused learning tool and the idea is that it's short sections which very briefly and concisely describe peering and different aspects of peering and the idea here is that it's a tool for ‑‑ these are people have been in the industry for a while or new people entering the industry, to discover what they need to do to follow the best practices and make their exhibitions for their network.
The beginners section is complete and the intermediary selection will be released in a couple of months. We are going to be making the website a little more interactive, it's very text‑heavy at the moment and we realise that he is not necessarily interesting to read or go through, we are looking at other ways to make the concept more interesting.
There's advice from experienced peers and we describe the use of IXP database and Peering DB.
The other reason that we actually started this project was because we have a number of networks and IXPs who are coming to us and saying, there's a lot of new networks that are joining IXPs and it's both the IXPs and the networks that are in a way hand‑holding the new entrants for a little while. So rather than having multiple people coming in and training or teaching, the idea of the Peering Toolbox is to provide a platform where everyone can just say go have a look here. And it's not something that Euro‑IX will not necessarily creating the content for this, we are collecting content from the community. There is a lot of good content out there that hasn't already been created and we want to put that in an easy to navigate formula for people to understand what they need to learn and happen.
In March this year ‑‑ well, we have a Route server Working Group, Stavros and we had a Route Server workshop in March. It was very successful, I'd like to say thank you to Namex again for hosting us, it was great to be in Rome for the workshop. We had about 20 IXPs present and there was lots of very interesting topics discussed. We also in these working workshops invite the developers who work on the root servers so we had a representative from BIRD, open BGP and Annika from Alice which is a looking glass.
We had Stavros presented what he has just presented now, regarding the Internet registry database and Route Server configuration and we had a really interesting presentation from Claudio for AS Pa, and Stavros talked about a Route Server wish list which is something we are looking on within environmental impact assessment.
We have the RFC8950 Working Group, I gave an update on what they worked on last year ‑‑ sorry, in Rome, since then there has been some more progress. They have done more Interop testing and the results are all published on GitHub, so if you are interested, if you find the Euro‑IX repository you can see the work going on there.
The supported implementation of RFC89 in root server and the team are developing a long‑term migration plan and idea. As Paul mentioned, we do have a Peering DB request for this work and the ticket number is 1576 so, like Paul said it would be really good to get some support from networks as well as the IXPs in this area.
What's next?
We are working on improving IXP manager and Alice‑looking glass. We are promoting and liaising with promoters and I want to specifically mention Meta here because I think they gave a presentation earlier on in the week and we would like to invite you to come and present what you are doing in this area at the next Euro‑IX meeting, so if anyone from Meta is here or listening please contact me.
And then large scale test scenarios are also happening and that's all available on the GitHub site.
Okay. So that's all in terms of the Euro‑IX update.
Now we are going to have a panel discussion on rules of engagement. So I would like to invite Remco and Paul on the stage. Thank you.
So, the idea of this panel discussion, and we do want it to be interactive, so it's not going to be me speaking to Remco and Paul; we want this to be interactive and we want your feedback as well because this is a topic that affects everybody so I ask that if you have any comments or anything to share on these topics, please come and ‑‑ please come to the mic and say what you have to say.
So, like I said, the idea of this panel is to discuss and look at how interconnection has evolved over the last decade or two. And are we still collaborating in the same way that we did ten years ago or are there too many barriers now?
So, the first question I have:
Over the past decade, how do you see the dynamics of peering have evolved and what impact has this had on the industry?
PAUL HOOGSTEDER: One of the things which I noticed over the last decade is the enormous explosion of the number of exchanges; whereas in the past you had one or two exchanges in a certain country or large region, but now we get loads and loads of both free exchanges, exchanges where you can get a 10 gig or even 100 gig port for free or for a minimal amount of money, and we see an explosion of distributed exchanges especially coming from Eastern Europe, that now expand into Western Europe and into the rest of the world with services. And this is a concern for people who want to keep their traffic local because they are into the, well, low latency business and do not want scenic routing which can easily be caused by this distributed exchanges.
REMCO VAN MOOK: Chiming in a little bit with what Paul said about the inflow of exchanges, what I notice is ‑‑ and this sort of plays into that thing ‑‑ I think in Amsterdam at this point I have lost count, but I think there's 28 internet exchanges in Amsterdam and part of that is because apparently there's some playbook out there, I am not entirely sure how, which makes running your own exchange part of your peering strategy. I am not sure that that is a sustainable model, at least not for the industry as a whole. So, right now, and this is also, I think, where the whole industry needs to take a look at themselves, right now people will go for oh, oh, they have a free 100 gig port, I will take that one, and oh, you now want some money, in that case I will go away again, which I think is leading to an ecosystem that is actually becoming increasingly under threat. And I do think as an industry as a whole, short term opportunistic behaviour is going to eat us all in the midterm, so please, all carefully consider what you are doing there.
Let's just leave it at that, that's a pretty cheerful thought to begin with.
BIJAL SANGHANI: Does anyone have anything to add or something to share on this?
WILL VAN GULIK: In this case you said traffic local. I think the definition of local traffic also tends to ‑‑ well, may have evolved with time. I mean, five milliseconds is something in Switzerland, go from one site to the other one and that is not going to happen in the Amsterdam region or something, that's also another change maybe or evolution in the industry, I am not sure.
REMCO VAN MOOK: I mean, I will add one more thing: I think what is now, it's a trend that started a decade ago is now a lot more prevalent, is ‑‑ is exchanges sort of increasingly becoming sort of the training wheels of peering environment, because as soon as traffic builds up to any significant volume, it will get killed off the exchanges and moved to AP and I somewhere, which I don't fault; I think it's actually ‑‑ it's a more cost efficient way of doing things, but it's also something to bear in mind because if you now look at traffic growth for exchanges, I mean the infamous up and to the right graphs that kills EIX ‑‑ is Nick in the room? I hope not. Please don't kill this Working Group yet. That is ‑‑ so, there is a lot less plastic visibility of what's going ‑‑ public visibility of what's going on and I am not saying that's a good or bad something but something I will definitely note.
AUDIENCE SPEAKER: So I am from OVH Cloud. I can relate to the distributed IXP major concern as well as remote peers, also, on some IXs. We have not found yet a way on our site to have a way to identify all of those outliers, talking about remote peers in that case, and find the correct policy we should have to have our traffic flow in the best way possible. So, yeah, that's a concern. It has to be addressed a.m. so the point to make some, I don't know, some standard way to identify outliers and so if there are remote ‑‑ distributed IXs in that place please give us all the tools with we need to be able to steer our traffic in the proper way if you want us to connect and have proper traffic flows
FLORENCE LAVROFF: Thank you.
AUDIENCE SPEAKER: Andrei Robachevsky. I am just can you remember no, sir because elsewhere in the Internet we observe a different trend of on sol addition and concentration which is built on network effect, I am just curious here we see an an opposite trend of decent /TRALising, what are your ideas and what are the forces behind?
REMCO VAN MOOK: That's actually an excellent question because I mean consolidation is order of the day in the whole network, in the whole Internet environment. If you look at the number of ISPs in any given European country, we have, I mean that's been cut from several dozen to a handful per country, of ISPs that have any significant size. So there's been a full on consolidation in the eyeball side of things, which actually those networks know and will use in their negotiating for interconnection, I am trying to keep this neutral.
At the same time, there's also a massive consolidation happening on the content traffic side of things and actually, there is ‑‑ the initial signs of this were a little bit concerning a decade ago where certainly the large mobile operators, I have heard a great example of a mobile operator in North America said well, you know, we will happily give room to a content inside of our data centres so they can just put their content basically inside of our network. Then ask how many of those content providers will you actually allow? Well obviously maybe five. Which then sort of allows those operators to essentially go and pick winners on the content side because no one else is going to be able to deliver content more effectively to that set of customers than the five that are already in there and that's something that I think we all should be wary of.
PAUL HOOGSTEDER: Agreed.
BIJAL SANGHANI: Anyone else have anything to say on this?
Next question for you both: In an industry where sharing internal knowledge can be fraught with legal and competitive risks, how can we foster a culture of openness and collaboration without come promising corporate interests?
REMCO VAN MOOK: I am looking at you. This is actually, in all ‑‑ in all frankness ‑‑ this has been something that Florence and I and, later, together with ‑‑ struggled mightily about in trying to content into this session. It is becoming increasingly complicated to have anyone representing any big logo to stand up and say anything and I don't blame those people for that. I also have no solution, because it really pains me that I know some of the smartest people in the industry are in this room and they will not say anything if their life ‑‑ even if their lives depended on it, not at a microphone, not in front of a camera, and I have no answer to that. I do think that this is ‑‑ we as a community, we are losing massively because of this.
PAUL HOOGSTEDER: Yeah, I do think this is mostly a problem for the bigger companies, the ones with lots of lawyers running around. Luckily working for smaller networks, I think most of the time they can speak freely, although even I can't name our customers.
BIJAL SANGHANI: Yes. Does anyone have anything to ‑‑
REMCO VAN MOOK: Please, I mean, this is a Meta comment, I think you are allowed Meta comments that are not related to company insights, I think, hopefully.
STAVROS KONSTANTARAS: Who had last year one of the biggest incidents in our company's history, where terabits of traffic went away. Anyway, there was a presentation, you guys know what happened, in the AMS‑IX culture and history we tried to be open and tried to share knowledge and experience with the community and yeah, it was not an easy presentation, for sure, but that's how we are, how we operate and I think we all need to the members because our customers trust us, they trust we do our best, they trust we do whatever we can do to make the network reliable and have them offering the best quality services but it's something that we also need to teach to your vendor, to the big logos, to the big enterprise, they are always afraid to take the blame, but hold on, you are the guy, you are the big vendor and the big logo who came to the RFP and you present your solution and you are trying to do to get any ‑‑ not AMS‑IX but any big network you try to get it as a customer, you should know that okay, no one has perfect solution, all the vendors have their own issues, so okay, make sure you offer to us the best quality services and then things like that will not happen in the future but also if things happen you should take your share, so it should be mutual agreement, not in contract but in understanding and culture. That's it.
BIJAL SANGHANI: Thank you. And thank you also I think for everyone on the presentation after the incident because I think it was interesting for everybody to learn what happened and how you overcame that so thank you for that, AMS‑IX.
Emile.
AUDIENCE SPEAKER: Emile Peterson. This openness is one of the reasons why I am in this industry, having some engineers that are willing to share, this is what ‑‑ this is the things we do, is why I chose this industry compared to other ones.
I am very much a fan of openness and I would love if more people would also be open. Thank you.
(Applause)
WILL VAN GULIK: Taking his company hat in this case. I have to say that lots of companies around have been really supportive and open and I will name people right now, so it's like free advertisement but Arista was someone who were really helpful even though I am just a grey market buyer. And I know N87 did help us a lot there and there is a bunch of people here in this crowd that were opened and that did help us to go forward and to understand what was going on.
So as I am part of this community because I discovered the openness in this crowd. Some discussion might have happened silently at the drink at some point and I think that was really, really useful. So I think we have got an openness, if we can keep it, that's going to be good. But we have to be careful and try to go that way and push that way and me as a small player I can maybe try to steer a bit in this way. I hope I can do it. So thank you for all the help, everyone.
BIJAL SANGHANI: Thanks, Will, keep up the good work.
AUDIENCE SPEAKER: Patrik speaking for Akamai technologies, trying to be present and speaking.
Two comments:
One, I think it's not just us trying not to talk, it's also us trying not to overwhelm and overtake the agenda so if we get guidance on what we should contribute, I am not just speaking for Akamai but content in general, I think that would be helpful and encouraged.
The second comment, and this is an unpopular one, I am fully acknowledging this and I don't have a good solution to it, one of the reasons why the majority of conversations happen in the floor and outside is because there is a complete difference on whether it's video‑recorded and live‑streamed or not. If we as a community could decide that we want to have a portion of the conversations on stage but not recorded and not live‑streamed, and again I acknowledged that is not inclusive and it is not for everyone else and I'm not saying it's a good solution but it's an idea we should talk about.
REMCO VAN MOOK: I am looking at the RIPE Chair right now because this ties in very closely to some conversations that we have had. I do agree with you that our current format actually in that sense sort of disencourages engagement and conversation. At the same time, I do also see that there's some very good principles about openness and transparency that we would like to try to honour but on the balance, if ‑‑ if that means we are not having conversation at all I think we are all losing.
AUDIENCE SPEAKER: Paul Rendek from DStream Group. You know, maybe a little portion of this is that this sector of our industry is maturing just like everything else has, and okay, we all probably came from that academic very open collaborative environment. Things have gone in all different directions for the IXP community, so to speak, right? And now when you start entering into this digital age that everybody likes to talk about, governments like to through at us, digital hubs or interconnection hubs, everybody wants to be one so I am not surprised I am see IXs coming in this because it seems to be part of the formula of how you would build a digital hub.
In the past I think these were kind of cities, the flaps just comes to mind, everyone wants a piece of that pie, which means that they think that in order to have this you probably have to have some kind of IX within your major city or even taking a look at it regionally now because that's what is starting to happen. Taking a look at Frankfurt or even London is kind of making us yawn a bit. I think the digital hub is maybe more looking at things regionally. So you probably would even see maybe there's a few dominant IXs in a larger city, but what's happening at the edge because everybody is taking a look at where that is, what is the relevance of this community, the IX community within that landscape? So when you look at openness and collaboration, what I'm maybe failing to see and I am not an expert here in this field but what are those criteria that you'd like to see that you think you can share and be collaborative on but I maybe probably saw that earlier on when the IX community came into the RIPE community but I am not sure if I am aware of what that is now so maybe we can put some of those on the table.
FLORENCE LAVROFF: Hi, Florence, no affiliation whatsoever. I would like to react to the culture of openness and collaboration since many of us are here thanks to that.
As Paul just said, we are maturing, this is what happens with the industry after some time but we still need to have new recruits, new people coming in and those people just like us, they need to come here and they need to manifest themselves and think with the community, thanks to those openness and corroboration principles that have attracted us so how can we maintain, like regardless of all those other corporate interests that may or may not be around us, how can we keep on having this openness and collaboration not only for you but for the one that needs to keep joining our ranks and for the ones that are going to be the community of the future?
REMCO VAN MOOK: Out of curiosity, in this entire room, who is a first time attendee? Can I have a show of hands. One, two ‑‑ welcome. I really should have done that at my opening. If you are new here, be welcome, talk to people, it is what we are here for, even if it's not recorded, even those people with the big logos on their badges are typically nice people and they will talk to you, as long as there's no cameras present.
AUDIENCE SPEAKER: Euro fibre. I want to like to hijack a little bit the topic here because like, just a few words why ‑‑ what I want to talk about. I was coming basically from the same environment everyone here is, I was working as service provider, working in IX. Now for the last three‑and‑a‑half years I was working with Eurofibre, Berlin, this is all connectivity, what I have noticed here is we were talking about going into connectivity in general, about openness and stuff and immediately deep diving into peering, IX and the other stuff.
So, what I am missing kind of here from my network experience now is, are there other forms of connectivity, other things how you connect? And especially in the access industry I notice in the last three‑and‑a‑half years, all the openness, all the discussion we have in this community here, this isn't taking place in the access community at all. And I wonder if this could also be a forum to collect some of those people around just to start the process because while the complaints are there, how did we change? Did we get less open and stuff like that? Just one more. We are in the IX community we are far farther than in all the other communities and in the Internet I have seen so far.
REMCO VAN MOOK: Actually, it's great that you bring this up. We rechartered a decade ago so this is no longer the IX community, this is why it's now called Connect and no longer EIX, and I do recall that we had a presentation, absolutely fantastic presentation, about someone building 25G fibre to the home connectivity in Switzerland, a few meetings back, and in that sense, speaking for the future and incoming Chairs because it's no longer my problem, but we would love to be spoiled for choice on topics that we present here. So, also to Patrik, don't worry about overloading us with material because you are really not, and as Chairs we would love to have a pick from all around the spectrum of interconnection and connectivity to present here.
AUDIENCE SPEAKER: Thank you.
AUDIENCE SPEAKER:
TORE ANDERSON: Bray. I wonder if lack of presentations is a good proxy for openness and collaboration. I understand you are challenged, I think the Working Group by nature regionally was a Working Group and not a conference, I would say what we heard today for instance from Stavros and the team that develops some new BCP, that is a good example of collaboration and I think maybe we need to be more clear that this is a Working Group and that we are not sort of looking really ‑‑ well, as well but not only for like you know professional presentations but also for like you come with a problem, problem statement is already good enough, maybe it's two slides but it might help.
BIJAL SANGHANI: It's important to have the discussions actually, and have a platform for them as well.
AUDIENCE SPEAKER: N T7, it wasn't me who presented about the 25 gigabits so ‑‑ it was Pascal our former CTO. But to follow up Marco's comment regarding openness in access networks, I mean, I just wondered do you really want to go into this field because it means a lot of regulatory, it means a lot of discussion and all that, because I mean we all know those which build fibre networks, they try to make it as close as possible because it means money, we are talking about money here, and I am not sure if this is the level on discussion we want to have in this Connect Working Group. So I could go further and talk about the so‑called glass ‑ in Switzerland, we had one recently but maybe you just read it first and then decide if you want to go into this field.
REMCO VAN MOOK: In all honesty, if I were still chair of the next meeting and someone submitted a presentation like that I would be over the moon, I would be ‑‑ extremely happy
AUDIENCE SPEAKER: I would volunteer for that, but I need at least 45 minutes to come to the basis of that stuff.
REMCO VAN MOOK: If that means we will have to extend connect to a second slot, I think we can have a discussion with the Working Group Chairs collective about that. If really, if there's really that much content I think we will find a way to put it somewhere. I do think it is important to keep ‑‑ because you are absolutely right, there is a whole bunch of cartel money, legal stuff that has come into this game, quite vividly, and but that's ‑‑ that doesn't mean it's a verboten topic or something. Actually, it's become part of the conversation. We are not no longer talking about banging rocks together, we are beyond that.
BIJAL SANGHANI: Thank you. Paul, do you have anything to add?
Any other comments? No. So I do actually have two more questions but we are, unfortunately, out of time. Where we do one more?
WILL VAN GULIK: We have got one minute and after that people will want to go to eat so I suspect we can shut this down now.
BIJAL SANGHANI: Okay. In that case, thank you. Thank you, Remco and Paul, for stepping up and being part of this panel and this discussion and also providing some excellent insight for hopefully everyone to go away and have a think about the different types of, well, rules of engagement in the community. So thank you.
(Applause)
WILL VAN GULIK: Thank you all, and with this, this session comes to an end and I would like to extend my thanks again to outgoing Chairs, thank you very much for what you did.
(Applause)
FLORENCE LAVROFF: Thank you.
WILL VAN GULIK: And reminder, rate the talks, and if you have any topics, you can contact the Chair collective and we will consider that for the next meeting that's happening in Prague. Thank you very much and bon appetit, see you next time.
LIVE CAPTIONING BY AOIFE DOWNES, RPR
DUBLIN, IRELAND
